Subset Pivotality in Sequential Multi-Arm Tests

Why the Global Null Is Not Enough by Itself

A sequential multi-arm test may examine several treatment arms at multiple interim stages. A boundary calibrated under the global null—that every treatment effect is null—can control the probability of at least one false rejection when that global null is true. This is called weak familywise error rate (FWER) control. Strong FWER control is more demanding: it requires the probability of rejecting at least one true null to be at most \(\alpha\) under every configuration of null and nonnull arms.

Subset pivotality is a distributional condition that can connect these two guarantees. It says, in essence, that the joint behaviour of the statistics belonging to true null hypotheses is not changed by effects in the other arms. When the condition holds, a global-null calibration of a boundary for the maximum across arms and stages also controls false rejections under partial-null configurations.

The result is useful only when the condition matches the actual design and test statistics. In particular, selection, shared controls, and changes to the sampling process can affect the joint distribution. A global-null calculation alone does not establish strong control.

Stagewise Statistics and Subset Pivotality

Suppose there are \(K\) treatment arms and \(L\) planned analysis stages. Let \(H_i\) be the null hypothesis for arm \(i\), and let \(Z_{i\ell}\) be its stagewise test statistic at stage \(\ell\). For concreteness, consider upper-boundary efficacy testing: arm \(i\) is rejected if its statistic crosses a prespecified upper boundary at an analysis when that arm is eligible for testing. The same reasoning applies to lower-tail or two-sided tests if their rejection events are included consistently in the calibration.

Definition: The stagewise statistics satisfy subset pivotality if, for every configuration of true and false arm-level null hypotheses, the joint distribution of the statistics for any subset of the true null hypotheses is the same as the distribution of the corresponding statistics under the global null, with nuisance parameters and design features held fixed. The subset includes the relevant stages for each arm. Thus, effects in arms outside the subset do not alter the joint null distribution for the subset.

This definition is about the joint distribution, not just the marginal distribution of each statistic. It therefore includes means, variances, and dependence across stages and arms. If the statistics are multivariate normal, for example, the mean vector and covariance submatrix for the true-null coordinates must agree with those under the global null. Equality of each statistic’s marginal standard normal distribution is not enough if the correlations change.

For sequential designs, the condition concerns the stagewise process used by the testing rule. If arms may be dropped, one can define the planned stagewise statistics as potential statistics, including those that would have been available had an arm remained under follow-up. The actual rule may observe only a subset of them. To use the result below, the rule must reject an arm only when its own eligible statistic crosses its specified boundary, and the global calibration must cover every arm-stage crossing that the rule could use. Subset pivotality of an unobserved potential statistic does not by itself justify a rule that changes thresholds in response to other arms’ outcomes.

A Model That Can Satisfy the Condition

A useful illustration is a joint-normal model for standardized treatment-control statistics. Let \(\delta_i\) denote the treatment effect for arm \(i\), measured in the outcome’s units, and let \(I_\ell\) be the information at stage \(\ell\). Suppose

Z_{i\ell}=\delta_i\sqrt{I_\ell}+\varepsilon_{i\ell}, \qquad \boldsymbol{\varepsilon}\sim N(\mathbf{0},\Sigma),

where the covariance matrix \(\Sigma\) is fixed across effect configurations. A null arm has \(\delta_i=0\), so its statistics have zero means at every stage. If changing effects in other arms leaves the relevant covariance submatrix unchanged, then the joint distribution for any set of null arms is the same as under the global null. This gives subset pivotality for that model.

The model also identifies a common failure mode. If treatment effects alter variances, allocation, or the correlation structure of null-arm statistics, then \(\Sigma\) may depend on which arms are nonnull. In that case, a null statistic can still have a standard normal marginal distribution while the joint null process differs from the global-null process. Subset pivotality must be checked for the joint stagewise distribution, not presumed from marginal validity.

Worked Example: Checking a Joint Null Subvector

Consider three arms and two stages, with statistics ordered as \((Z_{11},Z_{12},Z_{21},Z_{22},Z_{31},Z_{32})\). Suppose the model above holds with information levels \(I_1=25\) and \(I_2=100\), fixed covariance matrix \(\Sigma\), and effects \((\delta_1,\delta_2,\delta_3)=(0,0.04,0)\) outcome units per unit of the effect scale.

The mean vector is obtained by multiplying each arm’s effect by the square root of the stage information. Since \(\sqrt{25}=5\) and \(\sqrt{100}=10\), the means are

\boldsymbol{\mu} = (0,0,\;0.04(5),0.04(10),\;0,0) = (0,0,0.20,0.40,0,0).

Arms 1 and 3 are true nulls. Their four statistics have mean vector \((0,0,0,0)\) and covariance matrix equal to the submatrix of \(\Sigma\) indexed by \((Z_{11},Z_{12},Z_{31},Z_{32})\). Under the global null, this same mean vector and covariance submatrix apply, because \(\Sigma\) is assumed fixed. Thus the joint process for the true-null arms is unchanged by the positive effect in arm 2, and subset pivotality holds in this model.

If, instead, the effect in arm 2 changed the variance of a shared control outcome and thereby changed correlations involving arms 1 or 3, the relevant submatrix could differ. The zero means for the null arms would not be sufficient to establish subset pivotality.

Why Global-Null Calibration Can Give Strong Control

Let \(I_0\) be the set of true null arms under a particular configuration. Let \(R_i\) be the event that arm \(i\) is rejected, and let \(C_i\) be the event that at least one stagewise statistic for arm \(i\) crosses its boundary. The testing rule is assumed to satisfy \(R_i\subseteq C_i\): rejection requires a crossing by that arm’s own statistic. If an arm is dropped before a later stage, that can remove a possible crossing from the rule, but it cannot create a rejection without one.

The familywise error event is rejection of at least one true null. It is contained in the event that at least one true-null stagewise process crosses:

\{\text{at least one true null is rejected}\} \subseteq \bigcup_{i\in I_0} C_i.

Under subset pivotality, the probability of the right-hand event is the same as it would be for those same arms under the global null. That event, in turn, is contained in the global-null event that some arm crosses somewhere. If the maximum-boundary calibration makes the probability of that global event at most \(\alpha\), then

\begin{aligned} P(\text{at least one true null is rejected}) &\leq P\left(\bigcup_{i\in I_0} C_i\right)\\ &=P_{\mathrm{global\ null}}\left(\bigcup_{i\in I_0} C_i\right)\\ &\leq P_{\mathrm{global\ null}}\left(\bigcup_{i=1}^{K} C_i\right) \leq \alpha. \end{aligned}

This is the justification for strong FWER control. The first inequality uses the rule that rejection requires a crossing. The equality uses subset pivotality for the full stagewise processes of the true-null arms. The next inequality follows because the true-null arms are a subset of all arms. The final inequality is the global-null calibration. Selection does not invalidate this argument merely because it depends on interim data: it remains valid when selection can only restrict which own-statistic crossings lead to rejection and does not alter the calibrated boundary or the null processes.

Worked Example: Calibrating Six Independent Stagewise Statistics

Suppose, hypothetically, that three arms are tested at two stages, giving six independent standard normal statistics under the global null. Use a common upper boundary \(c\), and reject an arm if either of its two statistics exceeds \(c\). This independence assumption is only for transparent arithmetic; real sequential statistics are often correlated across stages and may be correlated across arms through shared controls.

Under the global null, each statistic has probability \(\Phi(c)\) of not exceeding \(c\), where \(\Phi\) is the standard normal cumulative distribution function. Independence gives

P_{\mathrm{global\ null}}(\text{no crossing})=\Phi(c)^6.

Set the probability of at least one crossing to \(0.05\). Then

1-\Phi(c)^6=0.05, \qquad c=\Phi^{-1}(0.95^{1/6})\approx 2.386.

Now suppose arms 1 and 3 are true nulls and arm 2 is nonnull. The four statistics for arms 1 and 3 remain independent standard normal under the assumed subset-pivotal model. Their probability of at least one crossing is

1-\Phi(2.386)^4 =1-0.95^{4/6} \approx 0.0336.

Thus the FWER for the two true-null arms is about \(0.0336\), below \(0.05\). The calculation illustrates the subset argument: under this model, the distribution of the four true-null statistics does not depend on the effect in arm 2.

Worked Example: Recalibrating for Four Independent Coordinates

In the same hypothetical independent setting, suppose a design has two arms and two stages, so there are four one-sided standard normal coordinates in the global maximum. To calibrate a global-null crossing probability of \(0.05\), solve

1-\Phi(c)^4=0.05, \qquad c=\Phi^{-1}(0.95^{1/4})\approx 2.234.

The calculation depends on having four independent coordinates. If the two stages within an arm are positively correlated, or the arms share a control, \(\Phi(c)^4\) is not the probability of no crossing. The threshold must instead be calibrated from the appropriate joint distribution, such as the specified multivariate-normal model. Subset pivotality does not supply that calibration; it allows a valid global-null calibration to extend to partial-null configurations.

Assumptions, Diagnostics, and Alternatives

Before relying on the result, verify the following features of the design and analysis:

Null-vector invariance: for each possible true-null subset, its joint stagewise distribution under any alternative configuration matches its distribution under the global null.
Correct dependence: calibration includes the across-stage and across-arm dependence, including any shared-control covariance.
Rejection requires a calibrated crossing: an arm cannot be rejected through a selection rule, altered threshold, or other statistic that was omitted from the global-null event.
Consistent process definition: arm dropping and changing information must not silently change the null process in a way not represented by the calibration.

A practical diagnostic is to compare the null-subvector distribution under several configurations: the global null and configurations with different nonnull effects in the remaining arms. In a normal model, compare the null-coordinate means and covariance submatrices. In simulations, generate repeated data under those configurations and compare the joint null-stagewise behaviour, including crossing probabilities. Agreement in a few marginal summaries is not proof, but disagreement is evidence that the condition or model needs revision.

Subset-pivotality-based calibration is often less conservative than Bonferroni control because it can use the actual dependence among arms and stages. Bonferroni control can remain available when valid marginal tests are available, without requiring the full joint subset-pivotal condition, but it may spend error inefficiently when statistics are strongly correlated. Another option is to calibrate directly under each relevant partial-null configuration; this can avoid relying on subset pivotality, but requires valid calculations across those configurations. These approaches answer different practical needs: global-null calibration is convenient when its distributional condition is credible, while direct partial-null analysis or a conservative bound may be preferable when it is not.

A common mistake is to argue, “The global-null probability of any crossing is \(0.05\), so the strong FWER is \(0.05\) under every configuration.” That conclusion does not follow from calibration alone. The correction is to establish subset pivotality for the joint true-null stagewise statistics and to check that every false rejection requires a crossing included in the calibration. If either condition fails, the global-null result may provide only weak control.

Key takeaway: Global-null calibration establishes strong FWER control when the joint stagewise distribution for every subset of true nulls is invariant to effects in the other arms, and the rejection rule is covered by the calibrated crossing event. Neither condition can be replaced by checking marginal null distributions alone.

Check Your Understanding

Answer each question, then compare with the short solutions.

In subset pivotality, why must the joint distribution be considered rather than only the marginal distribution of each null statistic?
A boundary controls the global-null probability of any crossing at \(0.05\). What additional distributional condition supports strong FWER control?
In the six-independent-statistic example, what is the calibrated one-sided boundary to three decimal places?
Why might shared controls make a calculation based on \(\Phi(c)^6\) inappropriate?
Can subset pivotality justify a rejection rule that lowers an arm’s boundary after observing another arm’s interim statistic? Explain.

Solutions. (1) The probability of any crossing depends on dependence as well as marginal distributions, so the joint law of all null-arm stages must be invariant. (2) Subset pivotality for every true-null subset, together with a rule in which rejection requires a crossing included in the global calibration. (3) \(c=\Phi^{-1}(0.95^{1/6})\approx 2.386\). (4) Shared controls can correlate arm statistics, so the probability of no crossing need not be the product of six marginal probabilities. (5) Not from subset pivotality alone: the changed boundary must be included in a valid calibration of the actual rule, or the event-inclusion argument may fail.