Sequential Bonferroni Tests For Multiple Endpoints

Why Allocate Error by Endpoint?

A trial may monitor several endpoint hypotheses at interim analyses. If each endpoint is tested at the full significance level, the chance of at least one false rejection can exceed the intended familywise level. Interim monitoring adds another opportunity for a false rejection unless the test for each endpoint already accounts for its repeated looks. A direct solution is to divide the total error budget among endpoints, then use a valid group sequential test at each endpoint’s assigned level.

The procedure is deliberately simple: allocate the global level in advance, do not transfer unused level between endpoints, and let each endpoint’s sequential test account for all of its planned looks. The endpoint tests can be dependent; the error-control argument does not require independence.

Fixed Endpoint Allocations and the Control Result

Choose nonnegative endpoint levels \(\alpha_1,\ldots,\alpha_m\) before examining the interim results, with \(\sum_{j=1}^m\alpha_j\leq\alpha\). For endpoint \(j\), use a group sequential test whose probability of ever rejecting \(H_j\), when \(H_j\) is true, is at most \(\alpha_j\). “Ever” includes rejection at any planned interim or final look. Each endpoint test must be calibrated for its own repeated looks; applying an ordinary single-look test repeatedly at level \(\alpha_j\) does not meet this condition in general.

Let \(R_j\) be the event that the procedure rejects \(H_j\) at any look, and let \(I_0\) be the set of indices of the true null hypotheses. A familywise error occurs when at least one of the events \(R_j\), \(j\in I_0\), occurs. The union bound gives the central justification:

The second inequality uses the validity of each endpoint’s group sequential test under its own null. The bound does not depend on the dependence among endpoints, nor does it require all null hypotheses to be true. It therefore establishes strong, rather than only global-null or weak, familywise error control.

A common allocation is equal Bonferroni allocation, \(\alpha_j=\alpha/m\). More generally, fixed nonnegative weights \(w_j\) with \(\sum_jw_j\leq1\) give \(\alpha_j=\alpha w_j\). These weights set priorities before the data are examined; this procedure does not recycle an endpoint’s unused level after that endpoint is rejected or reaches its final look.

How Sequential Testing Fits the Allocation

The outer allocation handles multiplicity across endpoints. Inside each endpoint, the group sequential test handles repeated looks. The local test must have its type I error controlled over the entire sequence of looks under the endpoint null. It may use boundaries calibrated for a specified spending function or another valid group sequential design; the particular boundary construction is separate from the endpoint allocation.

One transparent, generally conservative construction is to combine Bonferroni adjustment across looks with the endpoint allocation. Suppose endpoint \(j\) has \(K_j\) planned looks and a valid single-look p-value is available at each look. Test at look \(k\) only if its p-value is at most \(\alpha_j/K_j\). Under a true \(H_j\), each fixed-look p-value satisfies \(P(p_{jk}\leq t)\leq t\). Therefore:

This remains valid if the statistics at different looks are correlated, because the union bound does not assume independence. A group sequential design calibrated directly to \(\alpha_j\) can account for the joint distribution across looks and may be less conservative. Either way, the local requirement is the same: control the probability of any false rejection for endpoint \(j\) over its full monitoring sequence.

Unequal Allocations and Sequential P-Values

Equal allocation is not required. Investigators may assign more of the total level to endpoints considered more important, provided the levels are fixed in advance, nonnegative, and sum to no more than \(\alpha\). Allocation affects the local thresholds and hence the opportunity to reject each endpoint; it does not change the familywise guarantee if every local sequential test is valid at its assigned level.

A sequentially adjusted p-value for an endpoint summarizes the evidence along its monitoring path while accounting for the interim design. If \(q_j\) denotes such a p-value and is valid in the sense that \(P(q_j\leq t\mid H_j\text{ true})\leq t\), the endpoint can be rejected when \(q_j\leq\alpha_j\). It is not appropriate to substitute an unadjusted final-look p-value for \(q_j\) after repeatedly examining interim results.

Diagnostics, Alternatives, and Limitations

Before applying the procedure, verify both levels of calibration. Across endpoints, check that the exact assigned levels sum to no more than \(\alpha\). Within each endpoint, confirm that its test controls rejection over all planned looks at its assigned level, under the endpoint null and the actual monitoring plan. The endpoint test’s validity must reflect relevant design features, such as the analysis schedule and any adaptation that affects the test statistic or its distribution.

Warning signs include comparing every interim p-value with the full endpoint level, using a final-look p-value as though no interim analyses occurred, or rounding levels upward so their sum exceeds the global target. Another failure is using a local sequential procedure whose calibration assumes a different number or timing of looks. Bonferroni allocation cannot repair a locally invalid sequential test.

The method is easy to audit and remains valid under arbitrary dependence among endpoints, but it can be conservative. The union bound ignores the extent to which rejection events overlap, and a fixed allocation cannot make use of evidence from one endpoint to alter the level for another. The closure principle provides a more general framework for multiplicity control by testing intersection hypotheses; this simple allocation is attractive when a transparent, non-recycling rule is sufficient. A single global test may answer a different scientific question and, by itself, does not necessarily provide controlled endpoint-specific claims.

The procedure’s scope is also important. It controls the probability of one or more false endpoint rejections, not the probability that every endpoint decision is correct, and it does not guarantee adequate power for any particular endpoint. If endpoint hypotheses, weights, or the monitoring plan are changed in response to accumulating data, the original simple proof applies only if the revised procedure is itself shown to preserve the local error bounds and the total allocation.