Bonferroni Control and Its Guarantee

Why Control Error Across a Testing Family?

A genome-wide association study may test hundreds of thousands or millions of variants for association with a phenotype. Even when every tested null hypothesis is true, some small p-values can occur by chance. The relevant error question is therefore not just whether one test is calibrated, but how likely the analysis is to make at least one false rejection across the full family of tests.

Bonferroni control answers this question by assigning each test a more stringent threshold. Its main advantage is that its guarantee does not require the tests to be independent. To state that guarantee correctly, the family must first be specified: let $m$ be the fixed number of hypotheses included in the analysis, and let $\alpha$ be the desired upper bound on the family-wise error rate.

Definition: For a family of $m$ hypothesis tests, the family-wise error rate (FWER) is the probability of rejecting at least one null hypothesis that is in fact true. If $I_0$ is the set of true null hypotheses, then $$ \operatorname{FWER}=P\left(\bigcup_{i\in I_0}\{\text{reject }H_{0i}\}\right). $$ A p-value $p_i$ is valid for a true null hypothesis if $P(p_i\leq t)\leq t$ for every $0\leq t\leq1$. A continuous, correctly calibrated null p-value has equality; valid discrete or conservative p-values may have a smaller probability.

The family is the collection of hypotheses to which the error guarantee applies, not merely whichever results are eventually reported. In a GWAS, it might consist of the prespecified variant–phenotype tests in the analysis. The number $m$ is the total number of tests in that family, including tests whose null hypotheses may in fact be false. A careful definition matters: changing the family after examining results can make the stated error control misleading.

The Bonferroni Rule and Its Guarantee

For a target FWER of at most $\alpha$, the Bonferroni procedure rejects an individual null hypothesis only when its p-value is at most $\alpha/m$. Equivalently, it multiplies each raw p-value by $m$, caps the result at 1, and compares that adjusted p-value with $\alpha$.

Definition: For $m$ tests and target level $\alpha$, the Bonferroni threshold is $\alpha/m$. The Bonferroni-adjusted p-value for test $i$ is $$ p_{i,\mathrm{Bonf}}=\min(1,mp_i). $$ Reject $H_{0i}$ if and only if $p_i\leq\alpha/m$, equivalently if $p_{i,\mathrm{Bonf}}\leq\alpha$.

The equivalence follows directly: when $mp_i\leq1$, the adjusted p-value is $mp_i$, so $mp_i\leq\alpha$ exactly when $p_i\leq\alpha/m$. If $mp_i>1$, the adjusted p-value is 1 and cannot be at most an $\alpha$ between 0 and 1; also, $p_i>1/m\geq\alpha/m$. The cap keeps adjusted p-values in the usual interval from 0 to 1.

Theorem: Suppose the family contains $m$ tests, each p-value is valid whenever its null hypothesis is true, and the Bonferroni threshold is $\alpha/m$. Then the FWER is at most $\alpha$, under any dependence structure among the tests.

The result is often called strong FWER control: it applies for any configuration of true and false null hypotheses, not only when all null hypotheses are true. The proof uses the union bound, which says that the probability of a union of events is no greater than the sum of their probabilities.

Name the error events.
For each true null $i\in I_0$, let $A_i$ be the event that its null hypothesis is rejected. A family-wise error occurs if at least one of these events occurs.

Apply the union bound.
The probability of at least one false rejection is at most the sum of the probabilities of false rejection at the individual true nulls.

Use validity and the threshold.
At each true null, $P(A_i)=P(p_i\leq\alpha/m)\leq\alpha/m$. There are $m_0=|I_0|\leq m$ true nulls, so the sum is at most $m_0\alpha/m\leq\alpha$.

In symbols, the complete bound is

\operatorname{FWER} =P\left(\bigcup_{i\in I_0} A_i\right) \leq \sum_{i\in I_0}P(A_i) \leq \sum_{i\in I_0}\frac{\alpha}{m} =\frac{m_0}{m}\alpha \leq \alpha.

No independence assumption appears in this argument. The bound uses only validity of each p-value under its own true null and the fact that there are no more than $m$ true nulls in the family.

Worked Genome-Wide Calculations

Worked Example: Setting a Threshold for One Million Tests

Suppose a hypothetical GWAS tests $m=1{,}000{,}000$ variant–phenotype hypotheses and targets FWER $\alpha=0.05$. What is the Bonferroni threshold?

The model is a fixed family of one million tests with valid p-values for true nulls. The tests may be dependent; that does not invalidate the Bonferroni guarantee. The threshold is

\frac{\alpha}{m} =\frac{0.05}{1{,}000{,}000} =0.00000005 =5.0\times10^{-8}.

Thus a test meets this rule only if its raw p-value is at most $5.0\times10^{-8}$. For example, a hypothetical variant result with $p=4.6\times10^{-8}$ passes this threshold, while one with $p=5.4\times10^{-8}$ does not. The familiar $5\times10^{-8}$ value is the Bonferroni threshold for this particular choice of one million tests and $\alpha=0.05$; it is not a universal threshold for every testing family.

Worked Example: Comparing Adjusted P-Values

In the same hypothetical family of one million tests, two variants have raw p-values $p_1=4.2\times10^{-8}$ and $p_2=7.0\times10^{-8}$. Calculate their Bonferroni-adjusted p-values and decide which meets the $0.05$ FWER target.

For the first variant,

p_{1,\mathrm{Bonf}} =\min(1,1{,}000{,}000\times4.2\times10^{-8}) =\min(1,0.042) =0.042.

Since $0.042\leq0.05$, this result meets the Bonferroni rule. For the second variant,

p_{2,\mathrm{Bonf}} =\min(1,1{,}000{,}000\times7.0\times10^{-8}) =\min(1,0.070) =0.070.

Since $0.070>0.05$, the second result does not meet the rule. In this hypothetical analysis, the first test is significant by the Bonferroni procedure at FWER target $0.05$; the second is not. These adjusted p-values summarize the Bonferroni comparison with the full family. They are not probabilities that the corresponding null hypotheses are true.

Worked Example: A Smaller, Prespecified Family

A hypothetical study prespecifies $m=12{,}000$ variant tests for one phenotype and chooses $\alpha=0.05$. A selected test has $p=3.8\times10^{-6}$. Find the threshold and adjusted p-value, then state the conclusion.

The threshold is

\frac{0.05}{12{,}000} =4.1667\times10^{-6} \approx4.17\times10^{-6}.

The observed p-value is below this threshold. The adjusted p-value gives the same decision:

p_{\mathrm{Bonf}} =\min(1,12{,}000\times3.8\times10^{-6}) =\min(1,0.0456) =0.0456.

Because $0.0456\leq0.05$, the test passes Bonferroni control for the stated family. The interpretation depends on the family really being the 12,000 prespecified tests. If those tests were selected from a much larger set after looking at the data, this calculation would not by itself control error for the larger selection process.

Why the Bound Can Be Conservative

The union bound is an inequality, and it can be loose. It adds the individual probabilities of false rejection without accounting for how much the error events overlap. If two tests tend to produce small p-values together, their rejection events overlap; counting both event probabilities can overstate the probability of at least one rejection. In genetic association studies, correlation among variants can produce dependence among test statistics. Bonferroni remains valid in that setting, but its bound may be more conservative than the actual FWER.

Worked Example: Perfectly Duplicated Tests

Consider a hypothetical family of $m=4$ tests of the same true null, with identical, valid uniform p-values. The tests are perfectly dependent: all four p-values equal the same random variable $U$, where $U$ is uniform on $[0,1]$. At $\alpha=0.05$, the Bonferroni threshold is

\frac{0.05}{4}=0.0125.

A family-wise error occurs precisely when $U\leq0.0125$, because all four tests reject together or none do. Therefore the actual FWER is

P(U\leq0.0125)=0.0125.

The guarantee is satisfied, since $0.0125\leq0.05$, but the actual error rate is well below the target. The union-bound calculation only gives an upper bound of $4(0.0125)=0.05$; it does not claim that the FWER equals $0.05$. The example isolates why extensive overlap among rejection events can make the procedure conservative.

Conservatism is not the only practical issue. Bonferroni controls the chance of at least one false rejection, but it does not correct a p-value that is invalid because the association test is miscalibrated. Nor does a significant result establish that an association is causal, clinically important, or replicable. Those are separate questions. The guarantee is specifically about false rejections in the defined family, conditional on valid marginal p-values and the testing rule being applied as stated.

Common Mistake

Wrong reasoning: “The tests are correlated, so Bonferroni’s guarantee does not apply.”
Correction: The union-bound proof does not require independence. Dependence may make the bound loose and the procedure conservative, but it does not remove the guarantee when each true-null p-value is valid.
Wrong reasoning: “A Bonferroni-adjusted p-value of 0.04 means there is a 4% probability that the variant is a false finding.”
Correction: It is an adjusted measure used to decide whether the result passes the family-wise error rule. It is not the posterior probability that the null is true or that a particular finding is false.
Wrong reasoning: “The threshold is always $5\times10^{-8}$ in a GWAS.”
Correction: The threshold is $\alpha/m$. Its value depends on both the chosen family and the target $\alpha$; $5\times10^{-8}$ follows from $m=1{,}000{,}000$ and $\alpha=0.05$.

Key takeaway: For $m$ valid tests in a defined family, comparing every raw p-value with $\alpha/m$, or equivalently comparing its Bonferroni-adjusted p-value with $\alpha$, guarantees FWER no greater than $\alpha$ under arbitrary dependence. The guarantee is an upper bound and can be conservative.

Check Your Understanding

Use the family size and target level stated in each question.

For a family of 250,000 tests and $\alpha=0.05$, calculate the Bonferroni threshold.
State the condition on a p-value under a true null that is needed for the Bonferroni FWER guarantee.
Explain in one or two equations why Bonferroni control does not require independent tests.
For $m=20{,}000$, $\alpha=0.05$, and $p=2.0\times10^{-6}$, calculate the adjusted p-value and decide whether the test passes.
In the duplicated-test example, why is the actual FWER smaller than the union-bound upper limit?