Constructing Local Tests for Intersection Hypotheses

Why an Intersection Needs Its Own Sequential Test

In closed testing, an elementary arm-specific null hypothesis is rejected only when every intersection hypothesis containing it is rejected by its corresponding local test. The local test for an intersection must therefore control its own type I error, including the chance of a false rejection at any scheduled stage. Testing each arm at each look as though it were the only test does not generally meet this requirement: the probability of at least one crossing depends on the joint behavior of all the statistics in the intersection.

This tutorial constructs a local test for a specified intersection using its joint stagewise statistics. The task is to define the full rejection event, specify the null distribution of the statistics under the intersection, and choose a boundary so that the probability of that event is no greater than the local level \(\alpha\). Covariance matters: correlated statistics can have a different probability of any crossing than independent ones with the same marginal distributions.

Definition: Let \(I\) be a specified set of arms, with one-sided null hypotheses \(H_i:\theta_i\leq 0\). The intersection null is \(H_I=\bigcap_{i\in I}H_i\). A sequential local test of \(H_I\) is a rule based on the statistics observed at the scheduled stages that rejects \(H_I\) with probability at most \(\alpha\) whenever \(H_I\) is true.

For illustration, write \(Z_{ij}\) for the stagewise statistic for arm \(i\in I\) at stage \(j\), where \(j=1,\ldots,K\). Consider a one-sided crossing rule with fixed stage-specific boundaries \(c_j\): reject \(H_I\) if at least one statistic crosses its boundary. The event is

E_I=\bigcup_{i\in I}\bigcup_{j=1}^{K}\{Z_{ij}\geq c_j\}.

A common-boundary rule sets \(c_j=c\) for every stage. More elaborate rules may use different \(c_j\), but the calibration target remains the probability of the union of all crossing events, not the probability of a crossing at a single arm or stage.

Assumptions for Joint-Distribution Calibration

A convenient model is that the vector of all stagewise statistics for arms in \(I\) has a multivariate normal distribution with known covariance matrix \(R\), unit marginal variances, and means determined by the treatment effects. Under the least-favourable joint-null configuration, the vector is centred at zero. In the canonical setting, \(R\) includes both across-arm dependence, such as dependence induced by a shared control, and across-stage dependence from repeated looks.

\mathbf Z_I\sim N(\boldsymbol\mu_I,R),\qquad \mathbf Z_I=(Z_{ij}:i\in I,\ j=1,\ldots,K).

The joint-normal model and its covariance structure must be appropriate for the planned design. For a one-sided upper-crossing rule, suppose that under \(H_I\), each component mean is no greater than zero, and that \(R\) does not change across the null parameter values being considered. The crossing event is increasing in every component: increasing any \(Z_{ij}\) cannot turn a crossing into a non-crossing. Thus the probability of crossing is maximised at \(\boldsymbol\mu_I=\mathbf 0\). This justifies calibrating at the boundary of the intersection null under these assumptions.

If nuisance parameters change the covariance, or if the null permits mean patterns not bounded componentwise by zero, the zero-mean calculation may not be least favourable. In that case, validity requires controlling the rejection probability over the relevant null configurations, for example by calibrating the supremum rather than assuming the boundary configuration is worst. Subset pivotality and the joint-normal structure discussed in the earlier material can support such reasoning, but they do not excuse using a covariance matrix that does not match the specified intersection and design.

Calibrating the Boundary

For a common boundary, calibration chooses \(c\) to satisfy

P_{\boldsymbol\mu_I=\mathbf 0}\left(\max_{i\in I,\;1\leq j\leq K} Z_{ij}\geq c\right)=\alpha.

When the joint distribution is continuous, the crossing probability changes continuously with \(c\), so a numerical root-finding method can locate the cutoff. If the local test is intended to be conservative, the equality can be replaced with a probability no greater than \(\alpha\). The maximum is over every arm and stage belonging to this intersection. Leaving out an interim look or an arm changes the event and can invalidate the claimed level.

Worked Example: Calibrating a Two-Arm Intersection

Suppose the intersection is \(H_{\{A,B\}}=H_A\cap H_B\), with one planned look. Under its least-favourable null, let \(Z_A\) and \(Z_B\) have standard normal marginal distributions and correlation \(0.5\). Choose a common cutoff \(c\) so that the probability that either statistic crosses is \(0.05\).

Write \(\bar\Phi(c)=1-\Phi(c)\). Inclusion-exclusion gives the calibration equation

P(Z_A\geq c\ \text{or}\ Z_B\geq c) =2\bar\Phi(c)-P(Z_A\geq c,\ Z_B\geq c)=0.05.

The joint upper-tail probability is calculated from the specified bivariate normal distribution, not from an independence assumption. Numerically solving the equation gives \(c\approx1.916\). At this cutoff, \(\bar\Phi(c)\approx0.0277\), and the joint probability that both statistics exceed the cutoff is approximately \(0.0054\). Hence the union probability is approximately \(2(0.0277)-0.0054=0.0500\), up to numerical calibration error.

For hypothetical observed statistics \(Z_A=1.98\) and \(Z_B=1.72\), arm A crosses because \(1.98\geq1.916\); arm B does not. The local rule rejects the intersection because at least one statistic crosses. Under the stated joint-null model, the probability of this kind of rejection when the intersection is true is controlled at \(0.05\). This is a local intersection decision, not by itself a claim that every individual null in the intersection has been rejected by closed testing.

Justification: Why the Joint Crossing Probability Is the Target

A level-\(\alpha\) test must bound the probability of the event that makes the test reject. For a sequential local test, rejection occurs if any eligible arm-stage statistic crosses. Under the zero-mean joint-null model, setting \(c\) by the probability of this union to \(\alpha\) therefore gives a level-\(\alpha\) test at that configuration. Under the componentwise mean condition above, monotonicity extends the control to the rest of the intersection null.

The two-arm calculation illustrates why marginal cutoffs cannot be read as local cutoffs. If each of two statistics has marginal crossing probability \(q\), the union probability is \(2q-P(\text{both cross})\). The overlap term depends on their dependence. For more arms and stages, inclusion-exclusion quickly becomes cumbersome; evaluating the joint maximum distribution numerically is a standard way to calibrate the specified rule. The calculation must use the full covariance matrix for the statistics in the rejection event.

Worked Example: A Conservative Boundary Across Arms and Looks

Consider the intersection of two one-sided arm nulls assessed at two planned looks. Assume each of the four statistics has a standard normal marginal distribution under the intersection null. For this construction, no particular correlation structure is required, provided those marginal distributions and the null mean condition hold. Find a common boundary that guarantees local type I error no greater than \(0.05\).

There are \(mK=2\times2=4\) possible crossings. Set the upper-tail probability for each marginal statistic to \(0.05/4=0.0125\). The exact cutoff is \(c=\Phi^{-1}(0.9875)\approx2.241403\); using the slightly larger rounded cutoff \(c=2.242\) makes each marginal tail slightly less than \(0.0125\). For any dependence among the four statistics, the union bound gives

P(E_I)\leq\sum_{i\in I}\sum_{j=1}^{2}P(Z_{ij}\geq2.242) \approx4(0.01248)=0.0499<0.05.

Suppose the hypothetical observed values are \(Z_{A1}=1.90\), \(Z_{B1}=2.20\), \(Z_{A2}=2.05\), and \(Z_{B2}=1.60\). At the first look no statistic crosses. At the second look, none crosses either, so this local rule does not reject \(H_I\). If instead \(Z_{A2}=2.25\), the rule rejects at the second look because \(2.25\geq2.242\). This method is valid under its marginal assumptions, but it can be conservative because it ignores the overlap between crossing events. Exact joint calibration may yield a less conservative boundary when the joint distribution is reliably specified.

Intersection-Specific Calibration

A boundary belongs to a specified intersection and rejection rule; it is not automatically portable to another intersection. The number of arms, the number of included looks, their covariance, and any stage-specific boundary pattern all affect the probability of the union event. Even when two intersections have the same number of arms, their joint distributions can differ if the arms have different correlations or information patterns.

Worked Example: A Three-Arm One-Look Intersection

For a simple one-look illustration, suppose three statistics in \(H_{\{A,B,C\}}\) are independent standard normal variables under the joint null. Find the common cutoff for a local level of \(0.05\), then apply it to hypothetical data.

Independence means the probability that all three statistics fall below \(c\) is \(\Phi(c)^3\). Therefore the probability of at least one crossing is \(1-\Phi(c)^3\), and calibration requires

1-\Phi(c)^3=0.05,\qquad \Phi(c)=0.95^{1/3}\approx0.98305,\qquad c=\Phi^{-1}(0.95^{1/3})\approx2.1212.

For observed values \(Z_A=2.05\), \(Z_B=2.14\), and \(Z_C=1.66\), arm B crosses because \(2.14\geq2.1212\). The local test rejects the three-arm intersection. The independence assumption is essential to this particular equality: with correlated statistics, the cube \(\Phi(c)^3\) is not the probability of no crossing, and the cutoff must be recalculated using the actual joint distribution or replaced by a valid conservative construction.

Diagnostics, Alternatives, and Limitations

A calibration is only as trustworthy as its rejection event and model. Before using a boundary, check that the statistic vector includes every arm and scheduled look at which rejection is possible; that the marginal scale and one-sided direction are correct; and that the covariance matrix corresponds to the design, including shared-control and repeated-look dependence. Numerical calibration should also be checked by evaluating the crossing probability at the reported cutoff. A small rounding change can move the probability slightly, so report the cutoff to adequate precision and describe any numerical tolerance.

A useful diagnostic is to compare the calibrated joint crossing probability with both the individual marginal tail probabilities and a union bound. If the reported joint probability is smaller than a single included marginal crossing probability, the calculation or event definition is wrong. If a purported exact calibration simply multiplies marginal non-crossing probabilities despite correlated statistics, independence has been assumed without justification. If arm selection changes which statistics are tested, the unconditional event must reflect that selection rule; calibrating only the statistics that happened to be selected after observing the data can understate error.

The main alternatives are exact joint calibration under a credible joint model, a conservative bound such as the union-bound construction, or a different pre-specified sequential rule with its own calibrated rejection event. Exact calibration can use the dependence efficiently but is sensitive to model and covariance misspecification. A bound can remain valid without specifying dependence, but may sacrifice power. Neither approach automatically handles adaptation or changing hypotheses: those features require the actual testing procedure to be represented in the probability calculation. This tutorial’s construction assumes a fixed intersection and fixed, pre-specified boundaries.

Common Mistakes

Using a single-arm cutoff for the intersection. A marginal type I error does not equal the probability of at least one crossing across several arms and looks. Calibrate the union event.
Assuming independence because the marginal statistics are standard normal. Marginal distributions do not determine a joint crossing probability. Use the covariance structure or a dependence-robust bound.
Calibrating only at the final look. If an interim crossing can reject, omitting it from the event understates the rejection probability.
Reusing a boundary for a different intersection without checking it. The number of arms, stages, or dependence pattern may have changed. Verify the level for the new event.
Treating numerical calibration as exact. A rounded cutoff can produce a probability slightly above or below the target. Evaluate the probability at the reported value and retain sufficient precision in implementation.

Key takeaway: A sequential local test is defined by its entire arm-and-stage rejection event. To control its level, calibrate that event under a valid joint null model, justify why the calibration covers the composite intersection null, and verify the result using the same covariance and rule that will be used in the design.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For a local rule that rejects if any of three arms crosses at either of two looks, how many marginal crossing events belong in the union?
Why is \(P(Z_A\geq c\text{ or }Z_B\geq c)\) not generally equal to \(2\bar\Phi(c)\)?
Under what condition does a zero-mean calibration control the whole one-sided intersection null?
If a numerical cutoff gives a crossing probability of \(0.0506\) when the target is \(0.05\), is it level \(0.05\) as reported?
What is one advantage and one limitation of a union-bound cutoff compared with exact joint calibration?

Solutions. (1) There are \(3\times2=6\) arm-stage events. (2) The two crossing events can overlap, so their union probability is \(2\bar\Phi(c)-P(Z_A\geq c,Z_B\geq c)\); the overlap depends on the joint distribution. (3) The rejection event must be increasing in each statistic, the null means must be componentwise no greater than zero, and the relevant covariance structure must remain valid across the null configurations. (4) No. \(0.0506\) exceeds \(0.05\); the cutoff must be recalibrated or made more conservative. (5) A union bound avoids needing the dependence structure and guarantees control under its marginal assumptions, but it can be conservative because it ignores overlap among crossings.