Bonferroni Boundaries for Selected Arms

Why Fix the Boundaries Before Selection?

In a multi-arm multi-stage trial, interim results may determine which arms continue to later stages. That makes the set of tests actually performed data-dependent. A direct way to protect the familywise error rate (FWER) is to set boundaries for the entire planned collection of arm-stage tests before observing the data. Then, at each stage, apply only the boundaries corresponding to arms that remain eligible.

The key is that selection restricts which tests can reject; it does not create new tests with larger error budgets. The boundaries therefore remain valid even when the rule selecting arms depends on interim data, provided the individual test statistics have valid null distributions and the error allocation covers every possible arm-stage test. This guarantee can be conservative, but it does not require independence between arms or stages.

Here \(\Phi\) is the standard normal cumulative distribution function, so \(c_{ij}\) is the upper-tail critical value satisfying \(P(Z\geq c_{ij})=a_{ij}\) for \(Z\sim N(0,1)\). The rule is one-sided: large positive values provide evidence in the specified beneficial direction. If an application uses a different test statistic or a two-sided alternative, its critical values must be chosen to give the corresponding marginal tail probability no greater than the allocated budget.

Assumptions and Error-Control Argument

For the bound to apply, each planned test of a true null hypothesis must be marginally valid at its allocated level. In the normal case, this means that for every true null \(H_i\) and every stage \(j\), under the configuration of effects being considered, $$ P(Z_{ij}\geq c_{ij}) \leq a_{ij}. $$ Exact standard normal null distributions give equality for positive allocations; conservative null distributions give the inequality. Crucially, strong control requires this marginal validity for each true null under configurations in which other arms may have effects. Validity only under the global null establishes, at most, weak control. The earlier material on subset pivotality gives one standard condition under which marginal null distributions remain valid across configurations.

No independence assumption is needed. To see why, let \(\mathcal{I}_0\) denote the set of truly null arms. A familywise error occurs if at least one true-null arm is rejected at some stage. That event is contained in the union of all boundary-crossing events for true-null arm-stage pairs, whether or not those arms were selected. The union bound therefore gives:

The first containment is the important selection step: a rejection among selected arms is also a crossing among the full set of planned tests. The final inequality follows because the true-null arm-stage pairs are only part of the planned universe and all allocations are nonnegative. This proves strong FWER control, subject to the stated marginal validity condition. It also explains why an arbitrary dependence pattern does not break the guarantee: the union bound applies regardless of dependence.

Choosing the Allocation and Handling Zero Budgets

The total error budget can be divided across stages and then across arms within each stage. For example, choose stage budgets \(\alpha_1,\ldots,\alpha_J\) such that \(\sum_j\alpha_j\leq\alpha\), and choose arm budgets satisfying \(\sum_i a_{ij}\leq\alpha_j\) at each stage. This is a convenient design convention, not a requirement: any nonnegative fixed allocations whose grand total is at most \(\alpha\) meet the Bonferroni condition.

Allocations need not be equal. A design may assign larger budgets to stages or arms with greater scientific priority, but that decision must be made as part of the prespecified allocation. An arm-stage pair with \(a_{ij}=0\) is deliberately given no chance to reject under this rule. Its critical value is \(+\infty\), not a finite normal quantile. This convention keeps the rule defined for every allowed allocation and makes the consequence of a zero budget explicit.

Selection, Conservatism, and Alternatives

A common temptation is to divide a stage's budget only among the arms that survive to that stage. That changes the allocations according to interim data. The fixed-universe proof above no longer applies directly: it bounded the probability of each planned crossing by its fixed \(a_{ij}\), then summed those bounds. A data-dependent threshold can make the probability of crossing, together with being selected, larger than the planned marginal budget. Such reallocation might be justified by a method with an appropriate conditional-error argument, but it is not justified merely by the fixed Bonferroni inequality.

The price of fixed-universe Bonferroni is conservatism. The union bound can overstate the probability of a union when crossing events overlap substantially. For example, if six null statistics were perfectly correlated and each had a one-sided crossing probability of \(0.008333\), their joint familywise crossing probability would be \(0.008333\), while the Bonferroni bound would still be \(0.05\). At the other extreme, under independence the probability that at least one of six tests crosses is \(1-(1-0.008333)^6\), about \(0.04897\), much closer to the bound. These are dependence illustrations; the guarantee itself does not rely on either dependence pattern.

The practical alternatives have different requirements. Unadjusted armwise boundaries do not generally control FWER when several null hypotheses are tested. Joint-distribution calibration can use dependence information to obtain less conservative boundaries, but requires a suitable model for that dependence; the later material on Dunnett-type boundaries develops one shared-control approach. Bonferroni is attractive when robustness to dependence and a transparent guarantee matter more than maximizing power.

Diagnostics should focus on the assumptions and the implementation, not on whether the observed selection looks balanced. Confirm that the allocation sums to no more than \(\alpha\), that every planned arm-stage pair has a defined boundary, and that the correct boundary is used for each selected arm. Check that the null distribution used for each marginal test remains valid under configurations with non-null arms. In particular, reusing interim information in a later statistic may affect its null distribution; a standard normal marginal distribution should not be assumed without justification. If marginal validity fails, the union-bound calculation does not repair it.