Bonferroni Control From the Union Bound

Why a Per-Test Threshold Must Account for the Whole Family

In genomic analyses, a testing family may contain hypotheses about thousands of variants, genes, or other prespecified features. If each hypothesis is tested at the same nominal level used for a single test, the chance of at least one false rejection across the family can be much larger than that level. Bonferroni control gives a simple way to limit that chance without requiring independence among tests.

The key idea is to divide a total error budget across the hypotheses. The justification is the union bound: the probability that at least one of several events occurs cannot exceed the sum of their individual probabilities. This bound may be conservative, but it remains valid even when test statistics are strongly correlated, as is common for nearby genetic variants in linkage disequilibrium.

Definition: Consider a fixed family of $m$ hypotheses, with valid p-values $p_1,\ldots,p_m$, and let $I_0$ denote the indices of the hypotheses that are truly null. At a common threshold $c$, reject hypothesis $i$ when $p_i\leq c$. The familywise error rate (FWER) is the probability of rejecting at least one true null: $$ \operatorname{FWER}=P\left(\bigcup_{i\in I_0}\{p_i\leq c\}\right). $$ A p-value is valid under its null when $P(p_i\leq t)\leq t$ for every $t\in[0,1]$, whenever hypothesis $i$ is true.

The family has $m$ hypotheses, but only $m_0=|I_0|$ of them are true nulls. In practice, which nulls are true is unknown. Bonferroni uses the total family size $m$, which is known from the testing plan, rather than trying to estimate $m_0$. Its threshold for a target familywise level $\alpha$ is $\alpha/m$.

Deriving the Bonferroni Guarantee

Let $A_i=\{p_i\leq\alpha/m\}$ be the event that hypothesis $i$ is rejected using the Bonferroni threshold. A familywise error occurs when at least one of the events $A_i$ occurs for a true null. The union bound states that for any events $B_1,\ldots,B_k$, $P(\bigcup_{j=1}^k B_j)\leq\sum_{j=1}^k P(B_j)$. Apply it to the $m_0$ true-null rejection events:

\begin{aligned} \operatorname{FWER} &=P\left(\bigcup_{i\in I_0} A_i\right)\\ &\leq \sum_{i\in I_0}P(A_i)\\ &\leq \sum_{i\in I_0}\frac{\alpha}{m}\\ &=m_0\frac{\alpha}{m}\\ &\leq\alpha. \end{aligned}

The second inequality follows from marginal validity: for each true null, $P(p_i\leq\alpha/m)\leq\alpha/m$. The last inequality follows because $m_0\leq m$. If there are no true nulls, there cannot be a false rejection, so the FWER is zero.

This proof is the whole reason Bonferroni works. It does not factor a joint probability into a product, and it does not assume that tests are independent. It guarantees strong FWER control: the bound applies whatever the configuration of true and false null hypotheses. The earlier material on strong versus weak FWER control distinguishes this guarantee from control that holds only when all nulls are true.

Theorem: For a fixed family of $m\geq1$ hypotheses, if every true-null p-value is marginally valid, then rejecting $H_i$ whenever $p_i\leq\alpha/m$ gives $\operatorname{FWER}\leq\alpha$, for any dependence structure among the p-values.

Worked Threshold Calculations

The threshold is determined by the target $\alpha$ and the number of hypotheses in the family. It is applied to each hypothesis separately; it is not a threshold for an average or a sum of p-values.

Worked Example: A Genome-Wide Family of Variant Tests

Suppose a hypothetical study tests $m=20{,}000$ variant-level null hypotheses and seeks FWER control at $\alpha=0.05$. Assume each test produces a p-value that is valid under its null. The Bonferroni per-test threshold is

$c=\alpha/m=0.05/20{,}000=0.0000025=2.5\times10^{-6}$.

Suppose two hypothetical variants have p-values $1.8\times10^{-6}$ and $3.1\times10^{-6}$. Since $1.8\times10^{-6}\leq2.5\times10^{-6}$, the first hypothesis is rejected. Since $3.1\times10^{-6}>2.5\times10^{-6}$, the second is not rejected by this rule. Under the stated validity assumption, the probability of one or more false rejections among the 20,000 tests is at most 0.05, regardless of dependence among their test statistics.

This conclusion concerns the family as a whole; it does not mean that a particular rejection has a 5% probability of being false. Nor does it establish that a rejected variant has a causal effect.

Worked Example: Checking the Bound When Some Nulls Are True

Consider a hypothetical family of $m=1{,}200$ valid tests with target $\alpha=0.01$. The common Bonferroni threshold is $0.01/1{,}200=0.000008333$, or approximately $8.333\times10^{-6}$. Suppose, for illustration, that $m_0=900$ of these hypotheses are truly null. The union-bound calculation gives

\operatorname{FWER} \leq 900\left(\frac{0.01}{1{,}200}\right) =900(0.000008333\ldots) =0.0075.

Thus the bound in this particular configuration is 0.0075, which is below the target 0.01. The analyst generally does not know that exactly 900 nulls are true, so the procedure uses $m=1{,}200$, not 900, to set its threshold. Using $m$ ensures the guarantee even in the worst case $m_0=m$.

Unequal Error Budgets

The union-bound argument also allows different thresholds for different hypotheses. Choose nonnegative values $\alpha_1,\ldots,\alpha_m$ whose sum is at most the target $\alpha$, and reject hypothesis $i$ when $p_i\leq\alpha_i$. For every configuration of true nulls,

\operatorname{FWER} \leq\sum_{i\in I_0}P(p_i\leq\alpha_i) \leq\sum_{i\in I_0}\alpha_i \leq\sum_{i=1}^{m}\alpha_i \leq\alpha.

This allocation can reflect a prespecified analysis plan, but it does not create a larger total error budget. The thresholds must be chosen so their sum is no greater than $\alpha$, and each p-value must remain valid for its test.

Worked Example: Allocating a Familywise Level Across Four Hypotheses

Suppose a hypothetical study has four prespecified, valid tests and targets $\alpha=0.05$. Before examining the results, the investigator assigns thresholds $0.020$, $0.015$, $0.010$, and $0.005$, respectively. Their total is $0.020+0.015+0.010+0.005=0.050$, so the union-bound guarantee is at most 0.05.

Suppose the four hypothetical p-values are $0.018$, $0.016$, $0.009$, and $0.004$. Comparing each p-value with its own threshold, the first, third, and fourth hypotheses are rejected; the second is not, because $0.016>0.015$. The procedure controls the probability of at least one false rejection at no more than 0.05 under arbitrary dependence. This conclusion requires the allocation to have been specified independently of the observed p-values; assigning the largest thresholds after seeing which tests look most promising is not justified by this calculation.

Assumptions, Diagnostics, and Limits

Bonferroni makes a weak demand about dependence and an important demand about p-value validity. Dependence may be positive, negative, or otherwise structured. By contrast, if a true-null test is anti-conservative—so that $P(p_i\leq t)>t$ for some $t$—the central inequality in the proof can fail. For example, a poorly calibrated test or an analysis that ignores population structure may produce p-values that are not valid under the null. Reducing such p-values by a Bonferroni threshold does not, by itself, repair their invalid calibration.

The testing family must also be counted consistently. If the family contains all hypotheses for which the investigator intends a joint error guarantee, using only a subset of that count can make the threshold too large. When tests or a family are selected using the same data, validity may require accounting for that selection; treating a data-selected family as if it were fixed does not automatically preserve the stated bound.

Bonferroni can be conservative because the union bound does not account for overlap among rejection events. In highly correlated genomic tests, several events may tend to occur together, so adding their individual probabilities can overstate the probability of their union. Conservatism affects power, not the validity of the upper bound. A very small threshold may therefore yield few rejections even when some effects are real.

A useful diagnostic checklist is to verify the family size, the target level, the arithmetic, and the null calibration of each test. Also check that the threshold is compared with the p-value on the same scale and that two-sided versus one-sided testing was handled in the test that produced the p-value. Dependence is not a reason to reject Bonferroni’s guarantee; invalid marginal p-values or a misidentified family are.

The simplest alternative is to test every hypothesis at level $\alpha$, but that controls the error for an individual test, not generally the chance of any false rejection across the family. Bonferroni trades some potential power for a direct, assumption-light familywise guarantee. Other multiple-testing procedures can use additional structure or a sequential decision rule to improve power, but their guarantees and assumptions must be assessed separately; they do not follow merely from the Bonferroni derivation.

Common Mistakes

Assuming independence is required. The proof uses the union bound and marginal p-value validity, not independence. Correlation among tests does not invalidate the bound.
Dividing by the number of significant results. The threshold uses the number of hypotheses in the defined family, not the number that happen to look significant.
Applying $\alpha/m$ and claiming each test has level $\alpha$. Each true-null rejection probability is bounded by $\alpha/m$; summing across at most $m$ true nulls gives the familywise bound $\alpha$.
Interpreting $\alpha$ as the probability a particular rejection is false. FWER is a repeated-sampling probability of at least one false rejection in the family under the procedure’s assumptions; it is not a posterior probability for an individual result.

Key takeaway: If each true-null p-value is valid and the testing family has $m$ hypotheses, comparing every p-value with $\alpha/m$ guarantees $\operatorname{FWER}\leq\alpha$ under arbitrary dependence. The guarantee comes from the union bound, while its practical usefulness depends on defining the family and calibrating the tests correctly.

Check Your Understanding

Work through these questions, then compare your reasoning with the solutions.

A fixed family contains 500 valid tests, and the target FWER is 0.05. What common threshold does Bonferroni use?
Why does the Bonferroni guarantee not require independent tests?
For $m=80$, $\alpha=0.04$, and $m_0=50$ true nulls, calculate the union-bound upper bound on FWER using the common Bonferroni threshold.
Four valid tests are assigned thresholds $0.01$, $0.02$, $0.01$, and $0.01$. What familywise upper bound follows from the union bound?
What is wrong with using a Bonferroni threshold based on only the number of small p-values observed?

Solutions. (1) $0.05/500=0.0001$. (2) The union bound applies to any events: the probability of their union is at most the sum of their probabilities, whatever their dependence. (3) The common threshold is $0.04/80=0.0005$; therefore the bound is $50(0.0005)=0.025$. (4) The thresholds sum to $0.01+0.02+0.01+0.01=0.05$, so the FWER is at most 0.05. (5) The family size must come from the defined set of tests, not from their observed results. Using the number of small p-values makes the threshold data-dependent and does not establish the stated Bonferroni bound.