Bonferroni Control From the Union Bound in MAMS Trials

Why Divide the Significance Level?

A multi-arm trial can compare several treatments with a shared control. If each comparison is tested at the same nominal significance level, the chance of making at least one false rejection across the family can exceed that level. The tests may also be dependent because they use the same control group. Bonferroni control addresses both issues with a simple rule: divide the familywise significance level among the hypotheses in the prespecified family.

The rule is useful because it does not require the tests to be independent, nor does it require a particular pattern of correlation between treatment-control comparisons. Its price is that it can be conservative: it controls the error rate by bounding the chance of any false rejection, rather than calculating that chance exactly.

Setup and Assumptions

Suppose a trial has a prespecified family of \(m\) hypotheses, \(H_1,\ldots,H_m\), each tested using a p-value \(p_i\). The family is the set of hypotheses for which the trial will make a joint confirmatory claim. For example, it might consist of all treatment-versus-control comparisons for one primary outcome. The number \(m\) is fixed by that family definition.

Definition: The single-step Bonferroni procedure at familywise significance level \(\alpha\) rejects \(H_i\) when \(p_i \leq \alpha/m\). Equivalently, it compares every p-value with the same per-test significance level, \(\alpha/m\).

The essential assumption is that each p-value is valid for its own null hypothesis: whenever \(H_i\) is true, \(P(p_i \leq t) \leq t\) for every \(t\) between 0 and 1. This is the usual super-uniformity condition for a valid p-value. It includes exact uniform p-values under a continuous null distribution and conservative p-values, such as those from some discrete tests. In practice, validity depends on the model and test used for each comparison, including appropriate handling of the trial’s design.

No independence assumption is needed. In a multi-arm trial, comparisons with a shared control may be correlated; that correlation does not invalidate the Bonferroni guarantee as long as each individual p-value is valid. The family must also be specified appropriately: leaving relevant confirmatory hypotheses out of the family does not control error for claims that include them. This procedure concerns the defined family of tests, not repeated interim looks or other testing dimensions unless those tests are included in that family.

Deriving the Error-Control Guarantee

Familywise error occurs when at least one true null hypothesis in the family is rejected. Let \(I_0\) be the index set of the true null hypotheses, and let \(m_0\) be the number of elements in \(I_0\). Under Bonferroni, a true \(H_i\) is falsely rejected only if \(p_i \leq \alpha/m\). The probability of at least one such rejection is bounded by the sum of the individual probabilities.

\begin{aligned} P(\text{at least one false rejection}) &= P\left(\bigcup_{i\in I_0}\{p_i \leq \alpha/m\}\right)\\ &\leq \sum_{i\in I_0} P(p_i \leq \alpha/m)\\ &\leq \sum_{i\in I_0} \frac{\alpha}{m}\\ &= \frac{m_0\alpha}{m}\\ &\leq \alpha. \end{aligned}

The first inequality is the union bound: the probability of a union of events cannot exceed the sum of their probabilities. The second uses validity of each p-value under its true null. The last follows because \(m_0 \leq m\). If no null hypotheses are true, there can be no false rejection; otherwise the same bound applies.

This argument establishes strong familywise error control: the error rate is at most \(\alpha\) for any configuration of true and false null hypotheses, not only when every null is true. The proof does not calculate the exact familywise error rate. It supplies an upper bound that remains valid regardless of dependence among the tests.

Worked Example: Three Treatments Compared With a Shared Control

Consider a hypothetical trial comparing three treatments, A, B, and C, with one shared control on a prespecified primary outcome. The family contains the three treatment-control null hypotheses, so \(m=3\). Suppose the familywise level is \(\alpha=0.05\), and the valid p-values from the prespecified tests are 0.012, 0.041, and 0.200.

The equal per-test significance level is

\frac{\alpha}{m}=\frac{0.05}{3}=0.01667.

Treatment A has \(p_A=0.012 \leq 0.01667\), so its null hypothesis is rejected. Treatment B has \(p_B=0.041 > 0.01667\), and treatment C has \(p_C=0.200 > 0.01667\), so neither of those null hypotheses is rejected by this procedure.

The conclusion is that the trial rejects the treatment A versus control null at familywise level 0.05 under the stated assumptions. It does not mean there is a 5% probability that this particular rejection is false. The error-rate guarantee concerns the long-run probability of one or more false rejections across the family under repeated use of the procedure.

What the Bound Means in a Multi-Arm Trial

When every null in the family is true, \(m_0=m\), so the union-bound argument gives an upper bound of \(\alpha\). When some hypotheses are false, only the true nulls can contribute to familywise error, and the bound in the proof is \(m_0\alpha/m\). The procedure still uses the full family size \(m\) for its threshold; it does not need to know which hypotheses are true.

For shared-control comparisons, dependence can change the actual probability of at least one false rejection. But the union bound does not use the joint distribution of the p-values. It remains valid whether the tests are independent, positively correlated, or otherwise dependent, provided the marginal p-values are valid under their respective nulls. The guarantee may be conservative when the events of false rejection overlap substantially, because adding their probabilities can overstate the probability of their union.

Worked Example: Checking the Bound With Two True Nulls

Suppose a hypothetical family has \(m=4\) treatment comparisons and \(\alpha=0.05\). In one possible configuration, two null hypotheses are true and two are false. Under Bonferroni, each test uses

\frac{0.05}{4}=0.0125.

For either true null, validity implies that the probability its p-value is at most 0.0125 is no greater than 0.0125. Therefore, without assuming independence, the chance of falsely rejecting at least one of the two true nulls is bounded by

P(\text{at least one false rejection}) \leq 0.0125+0.0125 =0.025 \leq 0.05.

The two false nulls do not enter this error calculation: rejecting a false null is not a type I error. The bound is not an estimate of the exact familywise error rate; it is a guarantee that the rate is no greater than 0.025 in this configuration, and hence no greater than the target 0.05.

A Numerical Comparison With Unadjusted Tests

Testing each comparison at \(\alpha\) is not the same as testing the family at familywise level \(\alpha\). If there are several true nulls, each unadjusted test can contribute an opportunity for a false rejection. Bonferroni makes the per-comparison threshold smaller so the union of these opportunities is bounded.

Worked Example: Five Comparisons at Familywise Level 0.05

Suppose a hypothetical multi-arm trial has five prespecified treatment-control comparisons. Its valid p-values are 0.008, 0.013, 0.027, 0.044, and 0.110. With \(m=5\) and familywise \(\alpha=0.05\), the Bonferroni threshold is

\frac{0.05}{5}=0.0100.

Only the comparison with p-value 0.008 meets the threshold. The other four p-values exceed 0.0100. By contrast, using 0.05 as the threshold for each test would reject the first four null hypotheses, but that would not provide the same Bonferroni familywise guarantee. The difference is not evidence that any of the additional results are false; it is a difference in the decision rule and the error-rate guarantee attached to it.

Diagnostics, Alternatives, and Limitations

Bonferroni is straightforward to implement, but a valid result depends on details beyond the division itself. Before applying it, check that the family and its size were set consistently with the intended claims, that every p-value comes from a valid test of its stated null, and that each test uses the same threshold \(\alpha/m\). In particular, using a threshold based on fewer hypotheses than the family actually contains does not establish the stated guarantee for the full family.

Counting only the significant comparisons. The divisor is the size of the prespecified family, not the number of small p-values observed. Choosing the divisor after seeing results changes the rule.
Assuming shared-control dependence breaks the method. Dependence may affect the actual error probability, but the union-bound guarantee does not require independence.
Using unadjusted thresholds for a family claim. Testing every comparison at \(\alpha\) does not, by itself, control the probability of any false rejection across the family at \(\alpha\).
Assuming the correction repairs invalid p-values. If a test’s null distribution or model assumptions are unsuitable, dividing \(\alpha\) cannot make its p-value valid.

The main alternative to a single-step Bonferroni rule is a procedure that uses additional information, such as the ordering or dependence structure of the tests, to potentially gain power while controlling familywise error. Such methods require their own assumptions and justification. Another design decision is how to define the family, but that should follow the scientific claims and be prespecified—not be selected after inspecting the results. Bonferroni’s main limitation is potential conservatism, especially for a large family or strongly dependent tests: its simple upper bound may be far above the actual probability of any false rejection, reducing the chance of identifying genuine treatment effects.

Key takeaway: For \(m\) valid p-values in a prespecified family, rejecting when \(p_i \leq \alpha/m\) controls the familywise error rate at no more than \(\alpha\), under arbitrary dependence. The result follows directly from the union bound.

Check Your Understanding

Answer each question, then compare with the solutions.

A family contains six hypotheses and uses familywise \(\alpha=0.05\). What is the Bonferroni per-test threshold?
Why does the Bonferroni union-bound guarantee not require independent tests?
In a family of four hypotheses with \(\alpha=0.04\), two nulls are true. Give the union-bound upper bound on the probability of at least one false rejection.
Suppose a researcher applies a threshold based on three hypotheses but the prespecified family contains five. What is the problem?
Does Bonferroni guarantee that the probability of any false rejection is exactly \(\alpha\)? Explain.

Solutions. (1) The threshold is \(0.05/6=0.008333\) (approximately). (2) The union bound, \(P(\bigcup_i A_i)\leq\sum_iP(A_i)\), holds for dependent events as well as independent events; valid marginal p-values are sufficient. (3) The threshold per test is \(0.04/4=0.01\). For the two true nulls, the probability of at least one false rejection is at most \(0.01+0.01=0.02\). (4) Using \(0.05/3\), rather than \(0.05/5\), gives a larger threshold than the stated family warrants. The union-bound argument for the five-hypothesis family then does not guarantee control at 0.05. (5) No. Bonferroni guarantees that the probability is no greater than \(\alpha\); the union bound can be conservative, so the actual probability may be smaller.