Group-Sequential Boundaries Within Familywise Tests

Why Two Dimensions of Multiplicity Matter

A multi-arm trial may examine each treatment arm against a shared control at several planned interim analyses. This creates two distinct routes to a false positive: repeated opportunities to cross a boundary over looks, and multiple treatment hypotheses that may be rejected across arms. A boundary that handles one route does not automatically handle the other.

For example, a group-sequential boundary for arm A can control the probability of a false crossing over all its looks. Applying that same boundary separately to arms A and B does not, by itself, control the probability of falsely rejecting either arm. Conversely, a simultaneous boundary across arms at a single look does not automatically account for repeated looks. The design must address both dimensions in the local tests used for familywise inference.

Definition: Let \(H_i\) be the null hypothesis for treatment arm \(i\), and let \(H_I=\bigcap_{i\in I}H_i\) be the intersection hypothesis for a nonempty set of arms \(I\). At look \(k\), let \(Z_{i,k}\) be the standardized efficacy statistic for arm \(i\), oriented so that large positive values favour treatment. A local group-sequential test of \(H_I\) specifies a rejection event across the arms in \(I\) and all planned looks. It is a level-\(\alpha\) test if, whenever \(H_I\) is true, the probability of that event is at most \(\alpha\).

The familywise error rate (FWER) is the probability of rejecting at least one true elementary hypothesis \(H_i\). Strong FWER control means this probability is at most the specified level for every configuration of true and false elementary hypotheses, not just when all treatment hypotheses are true.

Writing a Local Test Across Arms and Looks

Suppose there are \(K\) planned looks. A simple local test for \(H_I\) uses an arm- and look-specific boundary \(c_{I,i,k}\). It rejects the intersection if at least one statistic for an arm in \(I\) crosses its boundary at any look:

R_I=\left\{\text{there exist } i\in I,\ k\in\{1,\ldots,K\} \text{ such that } Z_{i,k}\ge c_{I,i,k}\right\}.

A common special case uses a shared threshold \(c_{I,k}\) for all arms in \(I\) at look \(k\). The threshold sequence must be calibrated for the whole rejection event, including both all looks and all arms in the intersection. A local test is valid when

\sup_{\theta\in H_I} P_{\theta}(R_I)\le \alpha.

Here \(\theta\) denotes the parameters governing the joint distribution of the statistics. Under the usual canonical group-sequential model, the statistics across looks are jointly normal under the null, with variances standardized to one and correlations determined by the information levels; statistics for different arms may also be correlated, for example because they use a shared control. The joint covariance structure used to calibrate the boundary must match the design. If the model or nuisance-parameter assumptions do not support a valid upper bound on the crossing probability, the claimed local level is not established.

This is the key separation: the local test handles both dimensions together, while closed testing uses valid local tests for all intersections to obtain strong familywise control. A boundary calibrated only for one arm’s repeated looks is a within-arm sequential test, not automatically a valid local test for a multi-arm intersection.

Why Closed Testing Gives Strong Control

In closed testing, an elementary hypothesis \(H_i\) is rejected only if every intersection hypothesis containing it has been rejected by its own local test. Thus, for two arms, rejecting \(H_A\) requires rejecting both \(H_A\) and \(H_A\cap H_B\). This requirement is what allows local tests to support strong FWER control.

Theorem: If every nonempty intersection hypothesis \(H_I\) has a valid local test of level \(\alpha\), the closed-testing procedure controls the strong FWER at level \(\alpha\).

To justify the result, consider any configuration with at least one true elementary hypothesis. Let \(T\) be the set of all true elementary hypotheses. If closed testing rejects any true \(H_i\), where \(i\in T\), it must reject every intersection containing \(H_i\), including \(H_T=\bigcap_{j\in T}H_j\). Therefore,

\{\text{at least one true }H_i\text{ is rejected}\} \subseteq \{\text{the local test of }H_T\text{ rejects}\}.

Since every hypothesis in \(T\) is true, \(H_T\) is true. Its local test has rejection probability at most \(\alpha\). Taking probabilities in the subset relation shows that the FWER is at most \(\alpha\), for this configuration. The argument applies to every configuration of true and false hypotheses, establishing strong control.

Worked Example: A Sequential Test for Each Arm Is Not Yet a Familywise Test

Suppose two treatment arms are followed for three planned looks. For each arm separately, assume a group-sequential boundary has been calibrated so that, when that arm’s null is true, the probability of crossing at least one of its three boundaries is at most \(0.025\). In a hypothetical trial, the observed statistics are \(Z_{A}=(2.1,2.6,1.8)\) and \(Z_{B}=(1.7,2.0,2.2)\). Assume the calibrated boundary for each arm is crossed by A at look 2, but not crossed by B at any look.

The armwise tests reject \(H_A\) and do not reject \(H_B\). For any dependence between the armwise crossing events, the union bound gives

P(\text{at least one true arm is rejected}) \le P(\text{false crossing for A})+P(\text{false crossing for B}) \le 0.025+0.025=0.050.

Thus this particular allocation controls FWER at \(0.05\), provided each armwise crossing probability really is at most \(0.025\) under its null. The observed crossing for A is not evidence that the two armwise tests together had a \(0.025\) familywise level: the bound across arms is \(0.050\). Also, the armwise allocation can be conservative if the statistics are correlated. A directly calibrated local test for the intersection \(H_A\cap H_B\) may use that joint dependence, while still requiring valid local tests for the singleton hypotheses in the closure.

Worked Decisions Under Closed Testing

The following example illustrates why each intersection needs its own test spanning arms and looks. The boundary values below are hypothetical outputs of a valid joint calibration for the stated design; they are supplied for illustrating decisions, not as universal critical values. Assume two planned looks, efficacy is assessed in the positive direction, and crossing means \(Z\ge c\).

Worked Example: Combining Intersection Boundaries with Closure

For the singleton tests \(H_A\) and \(H_B\), suppose the calibrated look-specific boundaries are \(2.80\) and \(2.45\), respectively. For the intersection \(H_{AB}=H_A\cap H_B\), suppose the joint boundary is \(2.55\) at look 1 and \(2.35\) at look 2. Each local test is assumed calibrated to have level \(0.05\) over its full rejection event. The observed statistics are \(Z_A=(2.60,1.90)\) and \(Z_B=(1.80,2.50)\).

For the singleton test of \(H_A\), compare each look with its boundary: \(2.60<2.80\) and \(1.90<2.45\). It does not reject \(H_A\). For \(H_B\), the comparisons are \(1.80<2.80\) and \(2.50\ge2.45\). It rejects \(H_B\) at look 2.

For the intersection test, at look 1 the largest statistic is \(\max(2.60,1.80)=2.60\), and \(2.60\ge2.55\). The intersection test therefore rejects \(H_{AB}\) at look 1. Its look-2 statistics also include \(2.50\ge2.35\), but the first crossing already determines rejection; the second crossing is not a second independent test.

Under closure, rejection of \(H_B\) requires rejection of both \(H_B\) and \(H_{AB}\), which has occurred, so \(H_B\) is rejected. Rejection of \(H_A\) requires rejection of \(H_A\) and \(H_{AB}\); the singleton test of \(H_A\) did not reject, so \(H_A\) is not rejected. The conclusion is therefore evidence against the null for arm B, but not arm A, under the specified closed procedure. The example depends on the assumed validity of all the local calibrations; observing crossings alone does not establish their error rates.

Why Per-Look Error Rates Cannot Be Read as Sequential Error Rates

A frequent mistake is to treat a nominal level at each interim look as though it were the level for the entire sequential test. It is not. Even for one arm, the event of ever crossing is a union over looks, and its probability depends on the joint distribution across looks. Adding per-look error probabilities provides an upper bound, but it may be loose; ignoring the repeated opportunities provides no valid sequential guarantee.

Worked Example: The Cost of Ignoring Looks and Arms

Suppose there are two arms and three looks. If someone uses a \(0.05\) false-positive probability at each arm-look combination, the union bound for at least one false crossing is

P(\text{one or more false crossings}) \le \sum_{i\in\{A,B\}}\sum_{k=1}^{3}0.05 =6(0.05)=0.30.

This bound is not a claim that the actual error is \(0.30\); dependence between looks and arms affects the actual probability. It does show that the six nominal per-comparison levels alone do not guarantee a familywise level of \(0.05\). Even within one arm, three nominal \(0.05\) looks do not constitute a \(0.05\) group-sequential test.

A correction is to calibrate a repeated-look test for each arm at an allocated level and then control the across-arm error, for example by a Bonferroni allocation whose sum is at most \(0.05\). Alternatively, construct local tests for the intersections with boundaries calibrated jointly over all arms in the intersection and all looks, then apply closed testing. In either case, the error target applies to a defined rejection event, not to each isolated comparison.

Diagnostics, Alternatives, and Limitations

A useful audit starts with the event that the boundary is intended to control. For each local test, list the arms in the intersection and the planned looks, then verify that the calibration covers the probability of any crossing in that full set. Check that the stagewise covariance reflects repeated use of accumulating data and that the across-arm covariance reflects the control allocation and design. Confirm that every intersection needed by closure has a valid local test; a well-calibrated global-null test alone is not enough for strong control.

The principal alternatives differ in how they obtain valid local tests:

Armwise sequential tests with an allocation across arms control each arm’s repeated-look error and then use a bound such as Bonferroni across arms. This is transparent and robust to dependence, but can be conservative.
Intersection-specific joint boundaries calibrate the maximum over arms and looks within each intersection. They can reflect shared-control and repeated-look correlations, but depend on an adequate joint model and calibration.
A single simultaneous boundary applied across all arms and looks may be useful for a specified global test, but it does not automatically supply the local tests for all intersections that closed testing needs.

A boundary may also be poorly matched to the design if information levels, planned looks, or the covariance structure change. Reusing critical values after such changes requires a new justification. Finally, group-sequential boundaries address type I error for the specified efficacy rejection event; they do not by themselves determine stopping for futility, power, estimation after stopping, or how to handle changes to the hypotheses. Those are separate design questions.

Key takeaway: A repeated-look boundary controls a crossing probability over stages; a familywise procedure controls false rejections across arms. To obtain strong FWER control, use valid group-sequential local tests for every intersection and apply closed testing, or combine valid armwise sequential tests with an explicit across-arm error allocation.

Check Your Understanding

Answer each question, then compare with the solutions below.

One arm has a sequential test with false-crossing probability at most \(0.03\), and a second has one at most \(0.02\). What FWER bound follows from the union bound?
Why does a level-\(0.05\) global intersection test by itself not establish strong FWER control?
With three looks and a nominal \(0.04\) false-positive probability at each look, what union-bound guarantee follows for one arm’s probability of any false crossing?
In two-arm closed testing, the intersection test rejects and the singleton test of \(H_A\) does not. Can \(H_A\) be rejected? Explain.

Solutions. (1) The bound is \(0.03+0.02=0.05\), regardless of dependence, if the two armwise sequential tests have the stated crossing probabilities. (2) Strong control requires valid tests for every intersection, including intersections corresponding to subsets of the true nulls; a global test addresses only the intersection of all arms. (3) The union bound is \(3(0.04)=0.12\). This is an upper bound, not necessarily the actual crossing probability. (4) No. Closure requires rejecting every intersection containing \(H_A\), including the singleton \(H_A\) itself, as well as the two-arm intersection.