Bonferroni Adjustments for Selected Comparisons

Why Adjust Comparisons?

After an analysis of variance, a researcher may want to compare particular pairs of population means. But each additional test creates another opportunity to reject a true null hypothesis. Bonferroni adjustment controls this risk for a specified family of comparisons, whether the comparisons are all pairwise comparisons or only a selected set.

The method is straightforward: divide the desired familywise error rate across the comparisons. This tutorial shows how to adjust p-values and how to construct corresponding simultaneous confidence intervals. It also explains why the adjustment becomes more conservative as the family grows.

Definition: A family is the set of comparisons for which a researcher wants a joint error guarantee. If the family contains \(m\) comparisons and the target familywise error rate is \(\alpha\), the Bonferroni method tests each comparison at level \(\alpha/m\). For valid individual p-values \(p_1,\ldots,p_m\), the Bonferroni-adjusted p-value for comparison \(i\) is \(p_i^{\,B}=\min(1,mp_i)\).

The family must be specified clearly: the number \(m\) is the number of comparisons in the family, not automatically the number of groups in the study. For example, a study with four group means has six possible pairwise comparisons, but a researcher might specify a family of just three scientifically chosen comparisons. If the comparisons are chosen after inspecting the results, however, counting only the comparisons eventually reported may fail to account for the selection. The family should include the comparisons among which the researcher had the opportunity to choose.

Bonferroni-Adjusted p-Values

Suppose each comparison has a valid two-sided p-value calculated under its own null hypothesis. The Bonferroni decision rule rejects a null hypothesis when its individual p-value is at most \(\alpha/m\). Equivalently, multiply each p-value by \(m\), cap the result at 1, and compare the adjusted value with \(\alpha\).

p_i^{\,B}=\min(1,mp_i) \qquad\text{and reject }H_{0i}\text{ if }p_i^{\,B}\leq\alpha.

These two rules agree because \(mp_i\leq\alpha\) exactly when \(p_i\leq\alpha/m\). The adjusted p-value is not a new probability that the null hypothesis is true. It is a rescaled measure of evidence that supports the same familywise decision rule.

Worked Example: Adjusting Three Pairwise p-Values

Suppose a one-way ANOVA is followed by three specified pairwise comparisons among treatment means. Their valid two-sided p-values are 0.012, 0.031, and 0.200. Use a familywise significance level of \(\alpha=0.05\).

There are \(m=3\) comparisons, so the per-comparison cutoff is \[ \frac{\alpha}{m}=\frac{0.05}{3}=0.01667. \] The adjusted p-values are: \[ \min(1,3(0.012))=0.036,\quad \min(1,3(0.031))=0.093,\quad \min(1,3(0.200))=0.600. \]

Only the first adjusted p-value is at most 0.05. Thus, at the 5% familywise level, the evidence supports rejecting the null hypothesis for the first comparison, but not for the second or third. In particular, the second unadjusted p-value, 0.031, is below 0.05 but is not below the Bonferroni cutoff of 0.01667.

Why the Familywise Error Rate Is Controlled

The familywise error rate is the probability of making at least one Type I error in the family—that is, rejecting at least one null hypothesis that is true. Bonferroni control follows from a probability bound that does not require the test results to be independent.

Consider the true null hypotheses.
For each true null in the family, a valid test performed at level \(\alpha/m\) has probability at most \(\alpha/m\) of falsely rejecting it.

Bound the chance of any false rejection.
The probability of the union of the false-rejection events is at most the sum of their probabilities. There are no more than \(m\) true null hypotheses.

Apply the resulting bound.
The probability of at least one false rejection is therefore at most \(m(\alpha/m)=\alpha\).

P(\text{at least one false rejection}) \leq \sum_{i:\,H_{0i}\text{ true}}P(\text{reject }H_{0i}) \leq m\frac{\alpha}{m} =\alpha.

This is the union bound: the probability of a union of events is no greater than the sum of their probabilities. It remains valid when tests are dependent, as pairwise comparisons often are. Bonferroni therefore gives strong familywise error control, provided each individual p-value is valid when its null hypothesis is true.

Bonferroni Simultaneous Confidence Intervals

The same allocation of error applies to confidence intervals. To obtain a family of two-sided intervals with familywise confidence of at least \(1-\alpha\), construct each interval with marginal confidence level \(1-\alpha/m\). Each interval then has a noncoverage probability of at most \(\alpha/m\); the union bound limits the probability that one or more intervals miss their targets to at most \(\alpha\).

For comparisons of means in a one-way ANOVA, suppose the observations in group \(j\) follow a normal model with population mean \(\mu_j\) and a common variance \(\sigma^2\). Assume observations are independent, and that errors within and across groups are normally distributed with mean zero and the same variance. Let \(MSE\) be the pooled within-group mean square from the ANOVA, and let \(df\) be its error degrees of freedom. Under this model, \(MSE\) estimates \(\sigma^2\), and the usual pooled t interval for a pairwise mean difference has a t distribution with \(df\) degrees of freedom. These assumptions make the stated t-based coverage exact; if normality is only approximate, the coverage is approximate rather than guaranteed for a finite sample.

Formula: For \(m\) specified pairwise comparisons, a Bonferroni interval for the difference \(\mu_i-\mu_j\) is \[ (\bar{x}_i-\bar{x}_j)\ \pm\ t_{1-\alpha/(2m),\,df}\sqrt{MSE\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}. \] Here \(n_i\) and \(n_j\) are the group sample sizes. The critical value leaves probability \(\alpha/(2m)\) in each tail of an individual interval’s t distribution.

Worked Example: Intervals for Three Mean Differences

Suppose three groups each have \(n=8\) independent observations. A one-way ANOVA gives \(MSE=16\) and \(df=21\). Assume the exact normal-error model with a common variance described above. The sample means are 50, 45, and 43 points for groups A, B, and C, respectively. Construct Bonferroni intervals for the three specified differences A minus B, A minus C, and B minus C, using familywise confidence \(1-\alpha=0.95\).

There are \(m=3\) intervals, so the two-sided critical value is \[ t_{1-\alpha/(2m),df}=t_{1-0.05/6,21}=t_{0.991667,21}\approx 2.601. \] For each pair, the standard error is \[ SE=\sqrt{16\left(\frac{1}{8}+\frac{1}{8}\right)} =\sqrt{16(0.25)} =\sqrt{4}=2.000\text{ points}. \] The common margin of error is \(2.601(2.000)=5.202\) points.

The estimated differences are \(50-45=5\), \(50-43=7\), and \(45-43=2\) points. The intervals are therefore: \[ 5\pm5.202=(-0.202,10.202),\qquad 7\pm5.202=(1.798,12.202),\qquad 2\pm5.202=(-3.202,7.202). \] Rounded to two decimal places, these are \((-0.20,10.20)\), \((1.80,12.20)\), and \((-3.20,7.20)\) points.

Under the stated exact model and the other assumptions, the three intervals together have at least 95% familywise coverage for their specified population mean differences. The interval for A minus C excludes zero, indicating a positive difference for that comparison under this simultaneous procedure. The other two intervals include zero, so they do not establish a nonzero difference at this familywise confidence level.

How the Number of Comparisons Changes the Adjustment

The comparison count directly affects both forms of adjustment. With \(m\) tests, the individual testing threshold is \(\alpha/m\), and the adjusted p-value is \(mp_i\) (capped at 1). As \(m\) increases, a given raw p-value cannot become more significant under the Bonferroni rule. For intervals, increasing \(m\) reduces the allowed noncoverage probability \(\alpha/m\) for each interval. The required individual confidence level rises, so the critical value and interval width increase.

Worked Example: The Same p-Value in Families of Different Sizes

A comparison has raw p-value 0.009, and the target familywise level is 0.05. If the specified family contains four comparisons, its adjusted p-value is \[ \min(1,4(0.009))=0.036. \] This is below 0.05, so the comparison passes the Bonferroni rule. If the relevant family instead contains eight comparisons, its adjusted p-value is \[ \min(1,8(0.009))=0.072, \] which is above 0.05. Equivalently, the cutoffs are \(0.05/4=0.0125\) and \(0.05/8=0.00625\), respectively. The raw evidence has not changed; the larger family requires stronger evidence for a rejection.

The adjustment can be conservative because it controls the chance of any false rejection by adding individual error probabilities. That sum can exceed the actual probability of at least one false rejection, especially when rejection events overlap. A larger family generally widens Bonferroni intervals and lowers the chance of declaring any one comparison significant. This is the cost of a simple guarantee that does not depend on the tests being independent.

Common Mistakes and Practical Checks

Using the wrong number of comparisons. The divisor is the size of the family for which a joint guarantee is wanted. State which comparisons belong to that family before calculating.
Multiplying by \(m\) but forgetting the cap. An adjusted p-value cannot exceed 1, so use \(\min(1,mp_i)\).
Using a one-sided critical value for a two-sided interval. For \(m\) two-sided intervals, use \(t_{1-\alpha/(2m),df}\), with \(\alpha/(2m)\) in each tail.
Claiming exact coverage under merely approximate normality. The t intervals have the stated exact finite-sample coverage under the specified normal, independent, equal-variance model. Approximate normality supports only an approximate coverage claim.
Assuming tests must be independent. The union-bound justification works without independence. Dependence does not invalidate Bonferroni control when the individual p-values are valid.
Interpreting a confidence family as a probability about fixed means. The 95% statement concerns the long-run coverage of the procedure across repetitions under the model, not a 95% probability that fixed parameters lie in the already calculated intervals.

Key takeaway: Define the comparison family, count its \(m\) members, and apply the Bonferroni rule consistently. Adjusted p-values use \(\min(1,mp_i)\); simultaneous two-sided intervals use the critical value \(t_{1-\alpha/(2m),df}\). More comparisons make the procedure more conservative.

Check Your Understanding

Work through these questions, keeping the family of comparisons explicit.

A family contains five tests, and one raw p-value is 0.008. Find its Bonferroni-adjusted p-value and decide whether it meets a familywise level of 0.05.
Why does the Bonferroni familywise error guarantee not require independence among tests?
For four two-sided intervals with familywise confidence 95%, what marginal confidence level is assigned to each interval, and what tail probability is used on each side?
In a pooled one-way ANOVA interval for a difference of means, name the model assumptions needed for exact finite-sample t coverage.
A comparison’s p-value stays fixed while the defined family expands from three tests to six. Explain how its Bonferroni-adjusted p-value and decision can change.