Bonferroni Control for Selected Comparisons

Why Adjust for a Family of Comparisons?

A one-way ANOVA may motivate several specific questions: whether treatment A differs from B, whether C differs from a weighted average of A and B, or whether a control differs from selected treatments. If each question is tested at significance level \(\alpha\), the chance of making at least one false rejection across the set can exceed \(\alpha\). Bonferroni control addresses this by allocating the total error budget across a prespecified family of comparisons.

The method is useful for selected comparisons, including families that are not all pairwise comparisons. It applies whether or not the test statistics are independent. Its guarantee comes from the union bound, which can be conservative when the comparisons are strongly related.

Model, Notation, and Bonferroni Rule

Suppose there are \(k\) independent groups in a one-way fixed-effects model. For group \(i\), observation \(j\) is \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where the errors are independent and normally distributed with mean zero and common variance \(\sigma^2\). Thus, within each group the observations are normally distributed, and all groups share the same variance. Write \(n_i\) for the size of group \(i\), \(N=\sum_{i=1}^k n_i\), \(\bar{x}_i\) for its sample mean, and \(MSE\) for the pooled within-group mean square from the one-way ANOVA. Its error degrees of freedom are \(df_E=N-k\).

A mean comparison can be expressed as a contrast \(L=\sum_{i=1}^k c_i\mu_i\), where the fixed coefficients satisfy \(\sum_{i=1}^k c_i=0\). Its estimate and estimated standard error are

\hat{L}=\sum_{i=1}^k c_i\bar{x}_i, \qquad SE(\hat{L})=\sqrt{MSE\sum_{i=1}^k\frac{c_i^2}{n_i}}.

For example, \(\mu_A-\mu_B\) uses coefficients \(1,-1\) for groups A and B. A comparison of group B with the average of A and C is \(\mu_B-(\mu_A+\mu_C)/2\), with coefficients \(-1/2,1,-1/2\). The standard error formula accommodates unequal group sizes.

Definition: A prespecified family of comparisons is a fixed set of \(m\) hypotheses or corresponding mean contrasts chosen before examining the results used to test them. Bonferroni control tests each two-sided hypothesis at level \(\alpha/m\), or uses the critical value \(t_{1-\alpha/(2m),df_E}\). For a one-sided family, the corresponding per-test tail probability is \(\alpha/m\).

Why the Adjustment Controls Family-Wise Error

The family-wise error rate (FWER) is the probability of making at least one false rejection in the family. Let \(T\) be the set of comparisons whose null hypotheses are true. For each such comparison \(r\), let \(A_r\) be the event that its null hypothesis is falsely rejected. The event of at least one false rejection is the union of those events. The union bound states that the probability of a union is no greater than the sum of the individual probabilities. Since each test has Type I error probability at most \(\alpha/m\),

P\left(\bigcup_{r\in T} A_r\right) \leq \sum_{r\in T}P(A_r) \leq |T|\frac{\alpha}{m} \leq \alpha.

This argument does not require the tests to be independent. It also shows why the method controls FWER under any configuration of true and false null hypotheses: there can be at most \(m\) true nulls in a family of size \(m\).

For a single contrast under its null hypothesis, the statistic \((\hat{L}-L_0)/SE(\hat{L})\) follows a \(t\) distribution with \(df_E\) degrees of freedom under the stated model. The two-sided Bonferroni test rejects when its absolute value exceeds \(t_{1-\alpha/(2m),df_E}\). Equivalently, its ordinary two-sided \(p\)-value must be at most \(\alpha/m\).

Worked Tests for Selected Comparisons

Worked Example: Two Prespecified Pairwise Comparisons

Suppose a hypothetical one-way experiment has three independent groups, each with \(n_i=5\) observations. Assume the design and residual diagnostics are consistent with independent normal errors and a common variance. The group means are 42, 34, and 39 response units for A, B, and C. The pooled error mean square is \(MSE=16\), based on \(df_E=15-3=12\). Before examining the results, investigators specified the family \(\mu_A-\mu_B\) and \(\mu_A-\mu_C\), so \(m=2\). Use \(\alpha=0.05\).

For either pairwise difference, the estimated standard error is

SE=\sqrt{16\left(\frac{1}{5}+\frac{1}{5}\right)} =\sqrt{6.4}=2.530\text{ response units}.

For A minus B, the estimated difference is \(42-34=8\) response units, giving

t=\frac{8}{2.530}=3.162,\qquad df_E=12.

The ordinary two-sided \(p\)-value is approximately 0.0082. The Bonferroni threshold is \(0.05/2=0.025\), so this comparison is statistically significant under the family-wise procedure. For A minus C, the estimated difference is \(42-39=3\), giving \(t=3/2.530=1.186\) and an ordinary two-sided \(p\)-value of approximately 0.259. This does not meet the 0.025 threshold. The evidence therefore supports a difference between A and B in this family, but not between A and C.

Worked Example: A Selected Weighted Comparison With Unequal Group Sizes

In a hypothetical study, independent groups A, B, and C have sizes \(4,6,\) and \(5\), respectively, and means 20, 26, and 23 outcome units. Assume the one-way model conditions are reasonable. Here \(N=15\), \(k=3\), \(df_E=12\), and suppose the pooled error mean square is \(MSE=9\). The prespecified family contains three comparisons, one of which is the mean for B minus the average of A and C. For this comparison, the coefficients are \((-1/2,1,-1/2)\).

The estimated contrast is

\hat{L}=26-\frac{20+23}{2}=4.5\text{ outcome units}.

Because the group sizes differ, use each group’s own sample size in the standard error:

SE(\hat{L}) =\sqrt{9\left(\frac{(-1/2)^2}{4}+\frac{1^2}{6}+\frac{(-1/2)^2}{5}\right)} =\sqrt{9(0.2792)} =1.585\text{ outcome units}.

Testing \(H_0:L=0\) gives \(t=4.5/1.585=2.839\) with 12 degrees of freedom. The ordinary two-sided \(p\)-value is approximately 0.015. For the family of three, the Bonferroni threshold is \(0.05/3=0.0167\), so this comparison meets the adjusted criterion. The conclusion concerns B versus the average of A and C, not B versus each group separately.

Simultaneous Confidence Intervals

Bonferroni control also gives simultaneous confidence intervals for all contrasts in a prespecified family. For each contrast, use the same standard error as in its test and the common critical value \(t_{1-\alpha/(2m),df_E}\):

\hat{L}_r\ \pm\ t_{1-\alpha/(2m),df_E}\,SE(\hat{L}_r), \qquad r=1,\ldots,m.

Each interval separately has coverage at least \(1-\alpha/m\) under the model. By the union bound, the probability that one or more intervals in the family fail to cover their corresponding true contrasts is at most \(\alpha\). Therefore, the collection has simultaneous coverage of at least \(1-\alpha\). This is a statement about the coverage of the procedure over repeated sampling, not a probability assigned to a fixed parameter after observing an interval.

Worked Example: Four Simultaneous Intervals With Unequal Group Sizes

Consider a hypothetical one-way experiment with four independent groups A, B, C, and D. Their sample sizes are \(3,3,3,\) and \(4\), and their means are 12, 7, 10, and 8 measurement units. Assume independent normal errors with common variance are reasonable; the pooled error mean square is \(MSE=4\). Thus \(N=13\), \(k=4\), and \(df_E=13-4=9\). The prespecified family contains four pairwise comparisons: A minus B, A minus C, B minus D, and C minus D.

With \(\alpha=0.05\) and \(m=4\), the two-sided critical value is \(t_{1-0.05/(2\cdot4),9}=t_{0.99375,9}\approx3.111\). For A minus B, the estimate is \(12-7=5\), and

SE_{A-B}=\sqrt{4\left(\frac{1}{3}+\frac{1}{3}\right)} =\sqrt{\frac{8}{3}}=1.633, \qquad \text{margin}=3.111(1.633)=5.080.

Its simultaneous interval is \(5\pm5.080\), or approximately \((-0.080,10.080)\) measurement units. It includes zero, so the family-wise procedure does not establish a difference for A versus B.

For A minus C, the estimate is \(2\), and its standard error is also \(1.633\), giving the interval \(2\pm5.080=(-3.080,7.080)\). For B minus D, the estimate is \(-1\), and the unequal-size standard error is \(\sqrt{4(1/3+1/4)}=\sqrt{7/3}=1.528\). Its margin is \(3.111(1.528)=4.752\), giving \((-5.752,3.752)\). For C minus D, the estimate is \(2\) with the same standard error and margin, giving \((-2.752,6.752)\). All four intervals include zero. Taken together, the intervals have at least 95% simultaneous coverage for these four prespecified differences, subject to the model assumptions.

What the Guarantee Does—and Does Not—Say

The number \(m\) is the count of comparisons in the family being protected. It is not automatically the number of groups, nor is it necessarily the number of every comparison one could imagine. The comparison family should be defined before examining the outcomes used for inference. Choosing only the most striking results after seeing the data and then treating them as a small prespecified family does not preserve the stated guarantee.

Bonferroni can be conservative because it spends \(\alpha/m\) on each test even when the test statistics are dependent or when fewer than \(m\) null hypotheses are true. Its strength is its simplicity and its validity without independence among comparisons. If the family is large, the per-test threshold can be small and intervals wide; that reflects the more demanding goal of protecting the whole family rather than one isolated question.

Common mistake: “Each comparison uses a 0.05 test, so the family has a 0.05 error rate.” This is not generally true: testing several hypotheses at 0.05 can make the chance of at least one false rejection exceed 0.05. For a prespecified family of \(m\) comparisons, use \(\alpha/m\) for each test—or the corresponding Bonferroni critical value for intervals—to guarantee family-wise error no greater than \(\alpha\).

Check Your Understanding

Use the model assumptions and Bonferroni rules developed above.

A prespecified family has five two-sided comparisons and \(\alpha=0.05\). What is the per-comparison rejection threshold?
Explain why the union-bound argument does not require the comparison test statistics to be independent.
For a contrast with coefficients \((1,-1)\), unequal group sizes \(n_1\) and \(n_2\), and pooled error mean square \(MSE\), write its estimated standard error.
What does a 95% simultaneous confidence guarantee mean for a family of Bonferroni intervals?
Why can selecting comparisons after viewing the outcome data invalidate the family-wise guarantee based on the originally reported \(m\)?