Bonferroni's Inequality for Strong Error Control

Why Control Error Across a Family?

When several hypotheses are tested together, the relevant error may be making at least one false rejection, rather than making a false rejection on one particular test. For example, an analysis might examine several treatment contrasts and report every result passing a chosen threshold. A threshold that limits each test’s error separately does not, by itself, limit the chance of any false rejection across the family.

Bonferroni’s inequality addresses this problem without requiring independence or any other particular dependence structure among the tests. Its guarantee applies whether some tests are highly correlated, independent, or related in a more complicated way. The key condition is instead that each test’s rejection probability is controlled when its own null hypothesis is true.

Definition: Let $H_1,\ldots,H_m$ be a fixed family of null hypotheses, and let $A_j$ be the event that test $j$ rejects $H_j$. For a parameter value $\theta$, write $I_0(\theta)=\{j:\theta\in H_j\}$ for the indices of the true null hypotheses. The familywise error rate at $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta\!\left(\bigcup_{j\in I_0(\theta)} A_j\right). $$ A procedure has strong familywise error control at level $\alpha$ if $\operatorname{FWER}(\theta)\leq\alpha$ for every parameter value $\theta$. Weak control requires this bound only when all the null hypotheses are true.

Strong control matters because an analysis usually does not know in advance which hypotheses are true. A guarantee verified only under the global null may fail when some hypotheses are false and others true. Bonferroni’s argument covers every configuration of true and false nulls.

The Union Bound and the Bonferroni Guarantee

For any finite events $A_1,\ldots,A_k$, the union bound, also called Boole’s inequality, says that the probability of at least one event is no greater than the sum of their probabilities. It does not require the events to be independent.

Result: For any finite collection of events $A_1,\ldots,A_k$, $$ P\!\left(\bigcup_{j=1}^{k} A_j\right)\leq \sum_{j=1}^{k}P(A_j). $$ Proof: For each outcome, the indicator of the union is at most the sum of the event indicators: $$ \mathbf{1}\!\left\{\bigcup_{j=1}^{k} A_j\right\} \leq \sum_{j=1}^{k}\mathbf{1}\{A_j\}. $$ Taking expectations of both sides gives the stated inequality, since the expectation of an event indicator is its probability and expectation is additive for a finite sum. $\square$

To use this result for testing, suppose test $j$ has rejection event $A_j$, and choose nonnegative test levels $\alpha_1,\ldots,\alpha_m$ whose sum is at most the desired familywise level $\alpha$. The marginal validity condition is that, for every parameter value at which $H_j$ is true, test $j$ rejects with probability no greater than $\alpha_j$.

Theorem: Suppose that for every $j$ and every $\theta\in H_j$, $P_\theta(A_j)\leq\alpha_j$, where $\alpha_j\geq0$ and $\sum_{j=1}^{m}\alpha_j\leq\alpha$. Then the tests have strong familywise error control at level $\alpha$. Proof: Fix any parameter value $\theta$, and let $I_0(\theta)$ be its set of true nulls. Applying the union bound to only the rejection events corresponding to true nulls gives $$ \operatorname{FWER}(\theta) =P_\theta\!\left(\bigcup_{j\in I_0(\theta)}A_j\right) \leq \sum_{j\in I_0(\theta)}P_\theta(A_j) \leq \sum_{j\in I_0(\theta)}\alpha_j \leq \sum_{j=1}^{m}\alpha_j \leq\alpha. $$ The argument holds for every $\theta$, including parameter values at which only some of the nulls are true. Thus the control is strong. $\square$

The standard equal-allocation Bonferroni procedure takes $\alpha_j=\alpha/m$ for each test. Each individual test is then conducted at level $\alpha/m$. If $p_j$ is a valid p-value for $H_j$, this means rejecting $H_j$ when $p_j\leq\alpha/m$. Here, validity means that for every $\theta\in H_j$ and every $u\in[0,1]$, $P_\theta(p_j\leq u)\leq u$. Consequently, under $H_j$, its rejection probability is at most $\alpha/m$, as required by the theorem.

Worked Example: Two True Nulls With Disjoint Rejection Events

Suppose a family contains four tests and the target familywise level is $\alpha=0.05$. Bonferroni assigns each test level $0.05/4=0.0125$. Consider a parameter configuration where $H_1$ and $H_2$ are true, while $H_3$ and $H_4$ are false. Suppose the two true-null rejection events are disjoint and each has probability $0.0125$.

For example, take $U\sim\operatorname{Uniform}(0,1)$, and let $A_1=\{0\leq U<0.0125\}$ and $A_2=\{0.0125\leq U<0.025\}$. Each event has probability $0.0125$, and they cannot happen together. Therefore $$ \operatorname{FWER}=P(A_1\cup A_2) =0.0125+0.0125 =0.025. $$ The sum bound is exact here. It is below the target $0.05$ because only two of the four hypotheses are true; the false-null rejections do not count as familywise error.

When Is the Bound Conservative?

The union bound may overstate the probability of a union because it counts the probability of each event separately, including outcomes on which several events occur together. In fact, for a finite collection of events, equality holds precisely when the events do not overlap except on a probability-zero set. To see this, let $N=\sum_{j=1}^k\mathbf{1}\{A_j\}$. Pointwise, the difference between the sum of indicators and the union indicator is $N-\mathbf{1}\{N\geq1\}$, which is zero when $N\leq1$ and positive when two or more events occur. Its expectation is zero exactly when $P(N\geq2)=0$.

In multiple testing, Bonferroni can therefore be conservative for several reasons. Rejection events can overlap; the family may contain fewer true nulls than the total number $m$; or an individual test’s actual rejection probability under its null may be below its allocated level. The guarantee remains valid in each case, but the actual familywise error can be substantially smaller than the bound.

Worked Example: Perfectly Dependent Tests

Consider four valid p-values under a configuration where all four null hypotheses are true. Let $U\sim\operatorname{Uniform}(0,1)$, and set $p_1=p_2=p_3=p_4=U$. Each p-value is individually uniform, so each test is valid. At familywise level $0.05$, the Bonferroni threshold is $0.05/4=0.0125$.

All four tests reject together exactly when $U\leq0.0125$. Thus $$ \operatorname{FWER}=P(U\leq0.0125)=0.0125, $$ whereas summing the four marginal rejection probabilities gives $$ \sum_{j=1}^{4}P(p_j\leq0.0125)=4(0.0125)=0.05. $$ The rejection events are identical, not disjoint. Counting their probabilities separately makes the bound four times the actual familywise error in this example. No independence assumption is needed for the guarantee; this dependence pattern simply shows why it may be conservative.

Worked Example: Independent True-Null P-Values With Different Distributions

Take three tests with target familywise level $\alpha=0.05$, so the Bonferroni cutoff is $t=0.05/3=1/60$. Suppose $H_1$ and $H_2$ are true and $H_3$ is false. Under this configuration, let $p_1\sim\operatorname{Uniform}(0,1)$ and let $p_2$ have distribution function $P(p_2\leq u)=u^2$ for $0\leq u\leq1$; assume these two p-values are independent. The second p-value is valid because $u^2\leq u$ on this interval.

The first true null is rejected with probability $t=1/60$. The second is rejected with probability $t^2=1/3600$. Independence gives $$ \operatorname{FWER} =P(p_1\leq t\ \text{or}\ p_2\leq t) =t+t^2-t^3 =\frac{1}{60}+\frac{1}{3600}-\frac{1}{216000} =\frac{3659}{216000} \approx0.01694. $$ The union-bound calculation using the two true nulls gives $1/60+1/3600\approx0.01694$, slightly larger than the exact value because the events overlap. The simpler bound that uses all three allocated levels is $3/60=0.05$. Both bounds are valid; the latter leaves additional slack because $H_3$ is false and contributes no familywise error at this parameter value.

Common Mistake and Practical Interpretation

A common mistake is to argue, “Each test is run at level $0.05$, so the familywise error rate is at most $0.05$.” This confuses a marginal guarantee for one test with a guarantee about the union of several rejection events. The union bound only gives $P(\bigcup_{j\in I_0}A_j)\leq\sum_{j\in I_0}P(A_j)$; if every test is allowed a $0.05$ error probability, the resulting upper bound can be as large as $0.05m$, not $0.05$. For equal Bonferroni allocation, the correction is to use $0.05/m$ per test, which makes the sum across the full family no greater than $0.05$.

The other frequent error is to assume that the individual tests must be independent for Bonferroni control to work. Independence is not used in the proof. What is required is valid marginal error control for every true null, at the levels whose total does not exceed $\alpha$. Dependence affects how tight the bound is, not whether the stated guarantee holds.

Key takeaway: Bonferroni’s strong familywise guarantee follows by applying the union bound to the rejection events for the true null hypotheses. The argument is valid under arbitrary dependence, while overlap, conservative individual tests, and fewer true nulls can make the resulting bound conservative.

Check Your Understanding

Work through these questions using the definitions and argument above.

What is the difference between weak familywise error control and strong familywise error control?
For $m=8$ tests and target familywise level $0.04$, what equal Bonferroni cutoff is used for each valid p-value?
Why does the union-bound proof not require independence among test statistics or p-values?
Under what condition on a finite collection of rejection events is the union bound exact?
If only two of six null hypotheses are true, which rejection events enter the familywise error probability at that parameter value?

Bonferroni's Inequality for Strong Error Control

What You'll Learn