Two-Sided Endpoint Tests In A Multiplicity Graph

Why Two-Sided Tests Need Two Boundary Directions

A multiplicity graph allocates significance level across endpoint hypotheses. For a two-sided endpoint test, the allocated level is a total: it must cover evidence in both directions. At an interim look, a standardized statistic may cross an upper boundary, indicating an effect in the prespecified favorable direction, or a lower boundary, indicating an effect in the opposite direction. A graph allocation cannot be treated as the alpha available separately in each tail.

This distinction matters whenever an endpoint can be monitored more than once. The endpoint’s total local alpha must cover false rejection at any monitored look and in either direction. Thus the graph determines the endpoint’s total error budget; a tail-allocation rule and a sequential calibration determine how that budget is used by its lower and upper boundaries.

Definition: Let \(H_{0i}\) denote the null hypothesis for endpoint \(i\), and let \(\alpha_i\) be the total local significance level assigned to that endpoint by the multiplicity procedure. At look \(k\), let \(Z_{ik}\) be a standardized endpoint statistic oriented so that positive values favor the prespecified favorable direction. A two-sided sequential test has an upper boundary \(u_{ik}\) and a lower boundary \(l_{ik}\), with \(l_{ik}<u_{ik}\). It rejects \(H_{0i}\) at the first look \(k\) for which \(Z_{ik}\geq u_{ik}\) or \(Z_{ik}\leq l_{ik}\).

The boundaries are calibrated jointly across the looks for that endpoint. Under \(H_{0i}\), the calibration aims to keep the probability of crossing either boundary at any monitored look at or below \(\alpha_i\). The nominal local level is not a separate allowance to spend afresh at every look.

From Graph Allocation to Two Tails

A simple convention is equal-tail allocation. At a single look, it assigns \(\alpha_i/2\) to each tail. If the null distribution of \(Z_{ik}\) is standard normal, the boundaries are

l_{ik}=-z_{1-\alpha_i/2}, \qquad u_{ik}=z_{1-\alpha_i/2},

where \(z_q\) is the \(q\)-quantile of the standard normal distribution. With repeated looks, this single-look formula is not generally enough: the test must account for the chance of crossing at earlier looks as well as at the current one.

More generally, a design can prescribe a fraction \(q_i\) of the endpoint’s total alpha for the upper tail and \(1-q_i\) for the lower tail. Here \(0<q_i<1\). The cumulative upper- and lower-tail spending functions, written \(A_i^+(t)\) and \(A_i^-(t)\), describe how much error probability has been allocated by information fraction \(t\). They should satisfy

A_i^+(t)+A_i^-(t)\leq \alpha_i, \qquad A_i^+(1)+A_i^-(1)\leq \alpha_i.

For a fixed proportional split, the terminal targets are \(A_i^+(1)=q_i\alpha_i\) and \(A_i^-(1)=(1-q_i)\alpha_i\). Equal-tail testing uses \(q_i=1/2\). At each look, boundary calibration accounts for the joint distribution of that endpoint’s statistics across its looks and for crossings that would already have stopped the test. The two tails therefore share the endpoint’s budget; they are not calibrated as unrelated tests that each receive \(\alpha_i\).

Key distinction: The graph allocates a total local level \(\alpha_i\) to endpoint \(i\). The two-sided design allocates that total across upper and lower crossings, and sequential calibration allocates it across the scheduled looks. These are related but distinct steps.

Assumptions Behind the Boundaries

Boundary calculations require a specified statistical model and monitoring plan. A common group sequential model assumes that, under \(H_{0i}\), \(Z_{ik}\) has mean zero and variance one at each look, with information fractions \(0<t_{i1}<\cdots<t_{iK}=1\). Under a canonical normal approximation, the statistics across looks are jointly normal, and for looks \(j\leq k\),

\operatorname{Corr}(Z_{ij},Z_{ik})=\sqrt{\frac{t_{ij}}{t_{ik}}}.

This model represents accumulating information through independent increments. The actual information times, statistic definition, null distribution, and look schedule must match the design used to calibrate the boundaries. If the monitoring schedule changes, or the statistic does not have the assumed null behavior, the stated crossing probability need not hold. The endpoint-level calculation also does not, by itself, establish the joint behavior of different endpoints.

A prespecified sequential alpha-spending rule can be used to calibrate boundaries for the looks that actually occur. Its cumulative spending targets must be chosen so that the sum of upper and lower spending stays within the endpoint’s local level. A particular boundary pair is a result of calibration under the chosen model, not a universal consequence of knowing only \(\alpha_i\) and the look number.

Worked Examples

Worked Example: Splitting Graph Allocations at One Look

Suppose a hypothetical study has familywise significance level \(0.05\) and three endpoint hypotheses with initial graph weights \(0.50\), \(0.30\), and \(0.20\). The graph assigns local levels by multiplying each weight by \(0.05\). Assume one normally distributed test statistic per endpoint, a two-sided test, and equal-tail allocation.

The local levels are \(0.50(0.05)=0.025\), \(0.30(0.05)=0.015\), and \(0.20(0.05)=0.010\). Their sum is \(0.050\), as required by these initial weights. The per-tail levels are respectively \(0.0125\), \(0.0075\), and \(0.0050\).

For the first endpoint, the critical value is \(z_{1-0.0125}=z_{0.9875}\approx2.241\), giving boundaries \(-2.241\) and \(2.241\). For the second, \(z_{0.9925}\approx2.432\), giving boundaries \(-2.432\) and \(2.432\). For the third, \(z_{0.995}\approx2.576\), giving boundaries \(-2.576\) and \(2.576\).

If the observed statistic for endpoint 2 is \(-2.50\), it crosses the lower boundary because \(-2.50\leq-2.432\), so its local two-sided test rejects. This result is evidence in the negative direction under the stated orientation; it does not establish a favorable effect. Each calculation assumes the stated single-look normal model, and the graph’s use of these local tests must follow its prespecified error-control procedure.

Worked Example: Conservative Boundaries at Two Looks

Consider a hypothetical endpoint with local level \(\alpha_i=0.020\), monitored at two looks. Assume each null statistic has a standard normal marginal distribution, and use an equal-tail allocation at each look together with a Bonferroni construction across looks. No independence between the two looks is required for the bound below.

There are four possible tail crossings: lower and upper at each of two looks. Assign \(0.020/4=0.005\) to each. The standard normal critical value is \(z_{0.995}\approx2.576\), so use \(l_{i1}=l_{i2}=-2.576\) and \(u_{i1}=u_{i2}=2.576\).

At any one look, the probability of either tail crossing under the null is \(0.005+0.005=0.010\). Across both looks, the union bound gives

P_{H_{0i}}(\text{at least one crossing}) \leq 4(0.005)=0.020.

Thus the construction controls the endpoint’s false-rejection probability at no more than \(0.020\). For example, a statistic of \(2.60\) at the first look crosses the upper boundary and rejects the endpoint null. This procedure is valid under the stated marginal normal assumptions but may be conservative because it does not use the dependence between looks to calibrate less conservative boundaries.

Worked Example: Recalibrating After a Graph Transition

Suppose a hypothetical two-endpoint graph begins at familywise level \(0.05\), with weights \(0.60\) and \(0.40\). Endpoint 1 therefore starts with local level \(0.030\), and endpoint 2 with \(0.020\). Assume the prespecified graph transfers half of endpoint 1’s local level to endpoint 2 if endpoint 1 is rejected, with the other half leaving the active testing family.

After endpoint 1 is rejected, the transferred amount is \(0.50(0.030)=0.015\). Endpoint 2’s updated local level is \(0.020+0.015=0.035\). The total has not exceeded the original family level: \(0.035\leq0.050\). If endpoint 2 is tested once with a standard normal statistic and equal tails, its new critical value is \(z_{1-0.035/2}=z_{0.9825}\approx2.108\), rather than its initial \(z_{0.990}\approx2.326\).

The one-look value \(2.108\) is appropriate only for this example’s one-look normal test. If endpoint 2 has multiple looks, its upper and lower sequential boundaries must be recalibrated for the updated total local level and the prespecified tail-spending rule. Simply retaining the old boundaries would fail to reflect the new allocation.

Why the Calibration Matters

A short probability argument explains why both tails and all looks must be considered together. Let \(E_{ik}^+\) be the event of an upper crossing at look \(k\), and \(E_{ik}^-\) the event of a lower crossing. Rejection under \(H_{0i}\) occurs if any of these events occurs. Therefore,

P_{H_{0i}}(\text{reject }H_{0i}) = P_{H_{0i}}\left(\bigcup_{k=1}^{K}(E_{ik}^+\cup E_{ik}^-)\right).

A sequential calibration chooses boundaries so this probability is at most \(\alpha_i\), using the model for dependence across looks. A conservative alternative bounds the probability by the sum of the individual crossing probabilities. The bound may be larger than the actual probability because crossing events can overlap, but controlling that sum at \(\alpha_i\) ensures the endpoint’s false-rejection probability is no greater than \(\alpha_i\).

In a multiplicity graph, an endpoint’s two-sided local test is the test to which its allocated level applies. If the graph reallocates levels after a rejection, the endpoint’s total local level may change; the tail split and sequential boundaries must then be consistent with the updated level. The graph’s error-control guarantee also depends on using its prescribed testing and transition rules, not merely on checking that the initial weights add to one.

Diagnostics, Alternatives, and Limitations

Check the level being quoted. Confirm whether a reported alpha is the endpoint’s total local level or a per-tail amount. For an equal-tail two-sided test, each tail receives half of the total, not the full amount.
Check all looks and directions. A claimed sequential error probability must cover lower and upper crossings across the complete monitoring schedule, including the possibility of stopping early.
Check the null model and information times. Boundaries calibrated under a canonical normal model may not have their stated error rate if the statistic, information fractions, or look schedule differ materially from the design assumptions.
Check graph updates. After an endpoint rejection, verify the updated local level and whether remaining endpoint boundaries were recalibrated under the specified procedure.

Spending-based calibration uses the planned dependence across looks to allocate cumulative error and can be less conservative than a Bonferroni construction. Bonferroni is simpler and does not require independence across looks, but it can spend more conservatively because it bounds the probability of a union by adding crossing probabilities. Neither approach removes the need for a valid endpoint-level test or the graph’s required multiplicity rules.

A two-sided test is appropriate when departures in either direction count as rejection of the null. If the scientific claim is specifically about benefit, crossing a lower boundary may signal harm or an effect opposite to the target; it is not evidence of benefit. The interpretation and stopping action for either crossing should be prespecified. Finally, valid marginal boundaries for each endpoint do not alone describe dependence between endpoints or guarantee a particular joint probability of simultaneous crossings.

Common mistake: “The graph gives endpoint 1 alpha \(0.025\), so I can use \(0.025\) in each tail.” This uses \(0.050\) total for that endpoint before accounting for repeated looks. The correction is to treat \(0.025\) as the total local level, allocate it between tails, and calibrate the boundaries across all scheduled looks.

Check Your Understanding

Answer each question, then compare with the solutions below.

An endpoint receives local level \(0.018\). Under equal-tail one-look testing, what level goes to each tail?
Why does a two-look test with local level \(0.020\) require more than checking that each look individually has a two-sided error rate of \(0.020\)?
Under the Bonferroni construction in this tutorial, what two-sided critical value is used at each of two looks for local level \(0.020\)?
A statistic crosses the lower boundary. What does that establish, and what does it not establish about a favorable treatment effect?

Solutions. (1) Each tail receives \(0.018/2=0.009\). (2) Rejection can occur at either look, so the probability of at least one false rejection across the monitoring period can exceed the per-look probability; the total across looks must be controlled. (3) There are four tail-and-look opportunities, each assigned \(0.020/4=0.005\), so the critical value is \(z_{0.995}\approx2.576\), with boundaries \(-2.576\) and \(2.576\) at both looks. (4) It establishes rejection of the endpoint null by a statistic in the negative direction, under the specified test. It does not establish a favorable effect.