Bonferroni or Holm for Targeted Comparisons

Why Choose Between Bonferroni and Holm?

A one-way analysis may motivate a small number of focused questions: Does each of three treatments differ from a control? Do two selected treatments differ? When these questions form a family, testing each one at level \(\alpha\) without adjustment can make the chance of at least one false rejection larger than \(\alpha\). Bonferroni and Holm both control that family-wise error rate (FWER), but they use the family size differently.

Bonferroni uses the same, stricter threshold for every comparison. Holm orders the p-values and uses a sequence of thresholds, beginning with the Bonferroni threshold and then becoming less strict. As a result, Holm can reject more hypotheses than Bonferroni for the same family and significance level, while retaining FWER control.

The One-Way Model and the Comparison Family

For the usual one-way ANOVA setting, suppose there are \(k\) independent groups. In group \(i\), an observation \(X_{ij}\) follows the model

X_{ij}=\mu_i+\varepsilon_{ij}, \qquad \varepsilon_{ij}\sim N(0,\sigma^2),

where errors are independent, normally distributed, and have a common variance \(\sigma^2\). Thus the response distributions have group-specific means and a shared variance. The comparisons below use the pooled within-group mean square \(MSE\), with \(df=N-k\), where \(N\) is the total sample size. The model assumptions, and the validity of the tests that produce the p-values, matter: multiplicity adjustments do not repair dependent observations, severe model violations, or invalid individual tests.

A comparison family is the particular set of hypotheses to be considered together. For example, comparing three treatments with one control gives a family of three treatment-versus-control hypotheses. The family should reflect the scientific questions being addressed, not a selection made after inspecting which results look promising. Its size is denoted by \(m\).

Definition: The family-wise error rate is the probability of making at least one Type I error among the hypotheses in a specified family. Bonferroni and Holm procedures aim to keep this probability at or below the chosen significance level \(\alpha\).

A comparison can be a pairwise difference or a planned linear contrast. For a contrast with coefficients \(c_i\), where \(\sum_{i=1}^k c_i=0\), the estimated contrast and its standard error are

\widehat{L}=\sum_{i=1}^{k}c_i\bar{x}_i, \qquad SE(\widehat{L})= \sqrt{MSE\sum_{i=1}^{k}\frac{c_i^2}{n_i}}.

For a two-sided test of \(H_0:L=0\), the test statistic is \(t=\widehat{L}/SE(\widehat{L})\), compared with a \(t\) distribution with \(df=N-k\). Pairwise differences are a special case. Each valid individual test supplies a p-value; Bonferroni or Holm then adjusts the decision rule for the family.

Bonferroni and Holm Decision Rules

For Bonferroni, test each of the \(m\) hypotheses at level \(\alpha/m\). Equivalently, reject an individual null hypothesis when its p-value is no greater than \(\alpha/m\). This rule does not require the tests to be independent, but it can be conservative, particularly when the family is large or the comparisons are related.

Holm’s procedure begins by sorting the p-values from smallest to largest, \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). Compare them in that order. At rank \(i\), compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Continue while each p-value is at or below its threshold. At the first failure, stop: reject the hypotheses at the earlier ranks, but do not reject the hypothesis at the failed rank or any later rank. If every comparison passes, reject all \(m\) null hypotheses.

Decision rules: Bonferroni rejects comparison \(j\) if \(p_j\leq\alpha/m\). Holm sorts the p-values and tests \(p_{(i)}\leq\alpha/(m-i+1)\) sequentially, stopping at the first failure. Both procedures control FWER at level \(\alpha\) when the individual p-values are valid under their null hypotheses.

Holm’s first threshold is \(\alpha/m\), the same as Bonferroni’s. Its second threshold is \(\alpha/(m-1)\), and later thresholds increase until the last is \(\alpha\). The procedure is step-down rather than a collection of unrelated thresholds: a later hypothesis cannot be rejected if an earlier comparison fails.

Worked Example: Three Planned Comparisons With a Control

Suppose a hypothetical, balanced one-way study has four groups, eight observations per group, and \(MSE=16\) outcome-units squared. The total sample size is \(N=32\), so \(df=N-k=28\). The group sample means are 10.0 units for the control and 15.8, 15.0, and 14.4 units for treatments A, B, and C. The three pre-specified comparisons are each treatment versus control. Assume independent groups, a reasonable normal-error model, and similar within-group variances, as required by the one-way model.

For each treatment-control difference, the standard error is

SE=\sqrt{\frac{MSE}{8}+\frac{MSE}{8}} =\sqrt{\frac{16}{8}+\frac{16}{8}} =\sqrt{4}=2\text{ units}.

The estimated differences are 5.8, 5.0, and 4.4 units. Their test statistics are \(5.8/2=2.90\), \(5.0/2=2.50\), and \(4.4/2=2.20\). Using two-sided \(t\) tests with \(df=28\), the p-values are approximately 0.00720, 0.01855, and 0.03623, respectively.

At \(\alpha=0.05\), Bonferroni’s threshold is \(0.05/3=0.01667\). Only the first p-value is at or below 0.01667, so only the treatment A-versus-control null hypothesis is rejected by Bonferroni.

For Holm, the ordered p-values are already 0.00720, 0.01855, and 0.03623. The first threshold is \(0.05/3=0.01667\), and 0.00720 passes. The second threshold is \(0.05/2=0.02500\), and 0.01855 passes. The third threshold is \(0.05/1=0.05000\), and 0.03623 passes. Holm rejects all three null hypotheses. Both conclusions concern evidence of mean differences in this hypothetical setting; neither states that a particular treatment caused a change.

Why the Procedures Control Family-Wise Error

Bonferroni’s guarantee follows from the union bound. If \(A_j\) is the event that comparison \(j\) produces a false rejection, then

P\left(\bigcup_{j=1}^{m} A_j\right) \leq \sum_{j=1}^{m}P(A_j) \leq m\left(\frac{\alpha}{m}\right) =\alpha.

The first inequality holds for any dependence pattern among the tests. If a null hypothesis is true and its individual test is valid, testing it at level \(\alpha/m\) gives \(P(A_j)\leq\alpha/m\). Therefore the chance of at least one false rejection in the family is no greater than \(\alpha\).

Holm’s guarantee also holds without assuming independent tests. To see the logic, suppose there are \(m_0\) true null hypotheses, and consider the true-null p-value that appears earliest in the sorted list. Its rank is \(i\); at least \(m_0\) positions, including that rank and later ones, remain. If any true null is rejected, this earliest true-null p-value must pass its step-down test. Its threshold is \(\alpha/(m-i+1)\), which is no greater than \(\alpha/m_0\). Thus a false rejection requires at least one true-null p-value to be no greater than \(\alpha/m_0\). By the union bound, the probability of that event is at most \(m_0(\alpha/m_0)=\alpha\). This establishes strong FWER control: the guarantee applies whichever subset of the family’s null hypotheses is true.

Worked Example: Adding Comparisons Changes the Threshold

Return to the four hypothetical groups in the previous example. Suppose the scientific question instead calls for all six pairwise comparisons among the control and three treatments. This is a different family, with \(m=6\), not the original three treatment-versus-control comparisons. The three treatment-control p-values remain 0.00720, 0.01855, and 0.03623. The three treatment-treatment differences are 0.8, 1.4, and 0.6 units, giving test statistics 0.40, 0.70, and 0.30 with the same standard error of 2 units; their two-sided p-values exceed 0.05.

Bonferroni’s threshold for six comparisons is \(0.05/6=0.008333\). Of the three smaller p-values, only 0.00720 meets that threshold, so Bonferroni rejects only the treatment A-versus-control null hypothesis.

For Holm, the smallest p-value, 0.00720, is compared with \(0.05/6=0.008333\) and passes. The second-smallest is 0.01855; its threshold is \(0.05/5=0.01000\), so it fails. Holm stops there and rejects only the first null hypothesis. The added comparisons have changed the family size and the decisions, even though the original three comparisons’ p-values have not changed. This illustrates why the family should be specified by the questions of interest rather than expanded or narrowed after seeing results.

When Holm Gains Over Bonferroni

For the same family and p-values, Holm is never less permissive than Bonferroni. At every rank \(i\), its threshold \(\alpha/(m-i+1)\) is at least \(\alpha/m\). If a p-value passes Bonferroni, it also passes Holm: all preceding sorted p-values are no larger than it and therefore pass their Holm thresholds as well. Thus Holm rejects at least the hypotheses Bonferroni rejects, and may reject more.

Worked Example: A Step-Down Gain

Suppose four valid p-values for four pre-specified comparisons are 0.009, 0.016, 0.028, and 0.044, with \(\alpha=0.05\). These are hypothetical values from individual tests that meet the one-way model requirements. Bonferroni’s threshold is \(0.05/4=0.01250\), so it rejects only the comparison with p-value 0.009.

Holm first compares 0.009 with 0.01250; it passes. Next, 0.016 is compared with \(0.05/3=0.01667\); it also passes. The third p-value, 0.028, exceeds \(0.05/2=0.02500\), so Holm stops. It rejects the first two hypotheses and does not reject the third or fourth. In this example Holm gains one rejection without increasing the family-wise error rate above 0.05.

Choosing and Applying a Procedure

For a limited, scientifically targeted family, either method can be appropriate. Bonferroni is straightforward and gives one threshold to explain. Holm is often a useful default when several valid p-values are available: it retains the same FWER guarantee and can be more powerful, without requiring independent comparisons. The choice should be made in light of the questions and reporting plan, not by trying both procedures and presenting whichever gives more favorable results.

Define the family.
List the planned comparisons that belong together and state the chosen family-wise significance level \(\alpha\).

Check the individual tests.
Use the one-way model and appropriate degrees of freedom to obtain valid p-values for the specified comparisons.

Apply one family-wise rule.
For Bonferroni, compare every p-value with \(\alpha/m\). For Holm, sort the p-values and proceed step by step until the first failure.

Report the scope and result.
Name the family, method, significance level, and comparisons rejected. Interpret differences in the response’s units and in the study context.

Common Mistakes

Counting the wrong family size. The denominator is the number of comparisons in the defined family, not the number of groups. Three treatment-versus-control questions make \(m=3\); all pairwise comparisons among four groups make \(m=6\).
Continuing Holm after a failure. Holm is step-down. Once an ordered p-value exceeds its threshold, later hypotheses are not rejected, even if a later p-value happens to be smaller than a threshold considered in isolation.
Assuming Holm needs independent tests. Holm’s FWER guarantee does not require independence, though each individual p-value must be valid under its null hypothesis.
Choosing the family after seeing the p-values. Adding or removing questions based on results changes the inferential target and can undermine the intended error-rate control. Define and report the family transparently.
Calling non-rejection proof of no difference. A non-rejected null hypothesis is not established as true; it means the procedure did not find sufficient evidence to reject it at the specified family-wise level.

Key takeaway: Bonferroni compares every p-value with \(\alpha/m\). Holm starts with that threshold and then steps through increasingly permissive thresholds, stopping at the first failure. For the same defined family, Holm controls FWER and rejects at least as many hypotheses as Bonferroni.

Check Your Understanding

Try these questions using the decision rules and definitions above.

A family contains five planned comparisons and uses \(\alpha=0.05\). What is the Bonferroni threshold for each p-value?
For a Holm procedure with \(m=4\), write the ordered thresholds tested at ranks 1 through 4.
In a Holm procedure, the first ordered p-value passes, but the second fails. What can be concluded about the third and later hypotheses?
Explain why adding comparisons to a family can change adjusted decisions even when the original comparisons’ p-values stay the same.
Why can Holm reject more hypotheses than Bonferroni without giving up family-wise error control?