Why Holm Is Never Less Powerful Than Bonferroni

Why Compare Holm With Bonferroni?

A genomic analysis may test many hypotheses at once, such as whether each of several genes is associated with a phenotype. Controlling the familywise error rate (FWER) limits the probability of making at least one false rejection among the hypotheses in the family. Bonferroni and Holm both control FWER at a chosen level \(\alpha\), but they do not generally make the same decisions. The key comparison is exact: for the same p-values, family, and \(\alpha\), every hypothesis rejected by Bonferroni is also rejected by Holm.

This statement concerns the rejection sets produced from a given collection of p-values. It does not say that Holm must reject more hypotheses in every data set; sometimes their decisions are identical. It does say that Holm cannot discard a Bonferroni rejection. When Holm rejects additional hypotheses, the extra decisions arise from its step-down thresholds, not from a different choice of familywise error level.

Definition: Suppose a family contains \(m\) hypotheses with p-values \(p_1,\ldots,p_m\). Bonferroni rejects \(H_i\) when \(p_i\leq\alpha/m\). For Holm’s procedure, sort the p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\), carrying the corresponding hypotheses along. At rank \(j\), compare \(p_{(j)}\) with \(\alpha/(m-j+1)\). Reject sequentially while each comparison passes; at the first failure, stop and reject no hypothesis at that rank or later.

The equality convention matters at a boundary: a p-value equal to its cutoff passes the comparison. Tied p-values may be placed in any order among the tied hypotheses; their equal values are compared with the rank-specific Holm cutoffs. As in the earlier material on familywise error control, the family must be specified, and the p-values must be valid under their null hypotheses. Validity means \(P(p_i\leq t)\leq t\) for every \(0\leq t\leq1\) when \(H_i\) is true. No independence among tests is needed for the comparison or for the usual FWER guarantees of these two procedures.

The Rejection-Set Proof

The proof uses only the ordered p-values and the cutoffs. Let \(k\) be the number of Bonferroni rejections. Since Bonferroni rejects exactly the p-values at or below \(\alpha/m\), the first \(k\) ordered values satisfy \(p_{(j)}\leq\alpha/m\) for every \(j\leq k\). At each of those ranks, Holm’s denominator \(m-j+1\) is no larger than \(m\). Therefore its cutoff is at least as large as the Bonferroni cutoff:

\frac{\alpha}{m-j+1}\geq\frac{\alpha}{m} \qquad\text{for }1\leq j\leq k.

It follows that \(p_{(j)}\leq\alpha/(m-j+1)\) for every \(j\leq k\). Holm therefore passes each of the first \(k\) comparisons and rejects all those hypotheses. These are precisely the hypotheses with p-values at or below the Bonferroni cutoff, including any boundary ties. Holm may continue past rank \(k\), but it cannot stop before reaching it.

Theorem: For any fixed family of \(m\) p-values and any \(\alpha\), the set of hypotheses rejected by Bonferroni is a subset of the set rejected by Holm at the same \(\alpha\). This is a deterministic comparison of procedures on the observed p-values; it does not require independence.

The error-control assumptions explain why the comparison is useful. If each true-null p-value is valid, Bonferroni controls FWER because the probability of rejecting any true null is at most the sum of the individual rejection probabilities, which is at most \(m(\alpha/m)=\alpha\). Holm also controls FWER under arbitrary dependence. To see the logic, suppose there are \(q\) true nulls and consider the first true null in the ordered list. If Holm rejects any true null, it must pass the comparison at this first true-null rank \(r\). All \(q\) true nulls are at or after rank \(r\), so \(q\leq m-r+1\), and thus

p_{(r)}\leq\frac{\alpha}{m-r+1}\leq\frac{\alpha}{q}.

Consequently, the event of rejecting any true null implies that at least one of the \(q\) true-null p-values is at most \(\alpha/q\). By the union bound and p-value validity, that event has probability at most \(q(\alpha/q)=\alpha\). This brief justification also shows why neither procedure’s FWER guarantee requires independent tests.

Worked Decisions on Ordered P-Values

Worked Example: Holm Makes Additional Rejections

Suppose a hypothetical genomic screen tests \(m=5\) gene-phenotype hypotheses at familywise level \(\alpha=0.05\). The sorted p-values are \(0.001,\ 0.008,\ 0.013,\ 0.021,\ 0.049\). Assume the p-values are valid for their respective null hypotheses and that these five tests are the prespecified family.

Bonferroni. Its common cutoff is \(0.05/5=0.010\). The values \(0.001\) and \(0.008\) are at or below \(0.010\); the remaining values are larger. Bonferroni rejects the hypotheses at ranks 1 and 2.

Holm. Its successive cutoffs are \(0.05/5=0.010\), \(0.05/4=0.0125\), \(0.05/3\approx0.01667\), \(0.05/2=0.025\), and \(0.05/1=0.050\). The first comparison passes because \(0.001\leq0.010\). The second passes because \(0.008\leq0.0125\). The third passes because \(0.013\leq0.01667\). The fourth passes because \(0.021\leq0.025\), and the fifth passes because \(0.049\leq0.050\). Holm rejects all five hypotheses. In particular, \(0.040\) would also be below the fifth cutoff \(0.050\) if it were the fifth p-value; comparisons must be checked in the correct direction.

Here Holm retains both Bonferroni rejections and adds three. The extra decisions result because, after earlier ordered p-values pass, later Holm cutoffs become less stringent than the fixed Bonferroni cutoff.

Worked Example: A Step-Down Stop Limits Additional Rejections

Consider six hypothetical tests of genomic features with sorted p-values \(0.002,\ 0.009,\ 0.018,\ 0.021,\ 0.040,\ 0.700\), using \(\alpha=0.05\). Suppose, as before, the family is fixed and the p-values are valid under true nulls.

Bonferroni uses \(0.05/6\approx0.008333\). Only \(0.002\) is at or below that threshold, so it rejects rank 1.

For Holm, the first cutoff is \(0.008333\), and \(0.002\leq0.008333\), so rank 1 is rejected. The second cutoff is \(0.05/5=0.010\); \(0.009\leq0.010\), so rank 2 is also rejected. The third cutoff is \(0.05/4=0.0125\), but \(0.018>0.0125\). This is the first failure, so Holm stops and rejects no hypothesis at ranks 3 through 6. Holm makes two rejections, one more than Bonferroni, but it does not simply reject every p-value below some final common cutoff.

Worked Example: Identical Decisions at a Boundary

A hypothetical analysis tests \(m=4\) genomic hypotheses at \(\alpha=0.05\), with ordered p-values \(0.005,\ 0.0125,\ 0.030,\ 0.200\). Assume the same validity and family conditions as above.

Bonferroni’s cutoff is \(0.05/4=0.0125\). It rejects ranks 1 and 2; the second p-value is exactly equal to the cutoff, so it is rejected under the stated \(\leq\) rule.

Holm’s first cutoff is \(0.0125\), and \(0.005\) passes. Its second cutoff is \(0.05/3\approx0.01667\), and \(0.0125\) passes. The third cutoff is \(0.05/2=0.025\), but \(0.030>0.025\), so Holm stops. Its rejection set is also ranks 1 and 2. The theorem guarantees inclusion, not a strict increase in the number of rejections.

What “Never Less Powerful” Means

For the same data and testing family, Holm’s rejection set contains Bonferroni’s. In particular, for any individual false null hypothesis, the event that Bonferroni rejects it is contained in the event that Holm rejects it. Repeated over hypothetical repetitions of the study under the same model, this implies that Holm’s probability of rejecting that false null is at least Bonferroni’s. The same event-inclusion logic applies to outcomes such as rejecting at least one false null. This is the sense in which Holm is never less powerful.

The phrase does not mean that Holm’s findings are more biologically important, that every additional rejection is true, or that the realized number of rejections must be larger in every analysis. Nor does it justify changing the testing family after seeing the results. Both methods protect against false rejections within the family to which they were applied; scientific interpretation still depends on the hypotheses, effect estimates, study design, and measurement quality.

Diagnostics, Alternatives, and Common Pitfalls

A reliable comparison begins by checking that both procedures use the same \(m\), the same \(\alpha\), and the same p-values. Confirm that p-values are ordered from smallest to largest and keep each hypothesis label attached to its value. For Holm, verify the denominator at rank \(j\) is \(m-j+1\), and stop at the first p-value that exceeds its cutoff. A table of ranks, p-values, cutoffs, and decisions is a practical audit trail when a genomic analysis contains many tests.

Feature	Bonferroni	Holm
Comparison	Every p-value with \(\alpha/m\)	Ordered p-values with rank-specific cutoffs
Decision pattern	Reject each value at or below the common cutoff	Reject from the smallest upward until the first failure
Relative rejection set	Baseline single-step set	Contains the Bonferroni set for the same inputs
Dependence requirement for FWER control	No independence required if p-values are valid	No independence required if p-values are valid

A frequent mistake is to reason that Holm cannot improve on Bonferroni because both use the same \(\alpha\), or because Holm’s first cutoff equals Bonferroni’s. The first cutoff does match, but later Holm cutoffs increase: for example, at rank 2 the cutoff is \(\alpha/(m-1)\), which is greater than \(\alpha/m\). Passing earlier comparisons allows Holm to use those later, rank-specific cutoffs while preserving FWER control.

Another mistake is to continue after a Holm comparison fails, as if each later p-value could be tested independently against its own larger cutoff. That is not Holm’s rule: a failure ends the sequence. Finally, a computational mismatch may reflect different family sizes rather than different procedures. If one analysis counts 1,000 tests and another counts only 800, their Bonferroni cutoffs differ, and the rejection-set theorem cannot be applied as a same-input comparison.

Holm’s advantage is a guaranteed decision-set improvement over Bonferroni, not a guarantee of maximal power among all valid FWER procedures. Both can be conservative, especially when many tests are weakly informative. The appropriate method depends on the scientific aim and assumptions that can be defended; the present comparison establishes only the precise dominance of Holm over Bonferroni under identical inputs.

Key takeaway: If Bonferroni rejects \(k\) hypotheses, the first \(k\) ordered p-values are all at most \(\alpha/m\), and each corresponding Holm cutoff is at least \(\alpha/m\). Therefore Holm passes those \(k\) steps and cannot reject fewer hypotheses than Bonferroni on the same p-values.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For \(m=8\) and \(\alpha=0.04\), what are the Bonferroni cutoff and Holm cutoff at rank 3?
If Bonferroni rejects four hypotheses, what can be concluded about Holm’s decisions on the same p-values and at the same \(\alpha\)?
For \(m=5\), \(\alpha=0.05\), and ordered p-values \(0.004,\ 0.011,\ 0.014,\ 0.030,\ 0.080\), how many hypotheses does each method reject?
Does the dominance result require independent tests? What validity condition is relevant to the FWER guarantees?

Solutions. (1) The Bonferroni cutoff is \(0.04/8=0.005\). At rank 3, Holm’s cutoff is \(0.04/(8-3+1)=0.04/6\approx0.006667\). (2) Holm must reject all four Bonferroni-rejected hypotheses, and may reject additional ones. (3) Bonferroni’s cutoff is \(0.05/5=0.010\), so it rejects only \(0.004\). Holm’s cutoffs are \(0.010,\ 0.0125,\ 0.01667,\ 0.025,\ 0.050\). The first three comparisons pass: \(0.004\leq0.010\), \(0.011\leq0.0125\), and \(0.014\leq0.01667\). The fourth fails because \(0.030>0.025\), so Holm rejects three. (4) No independence is needed. Under each true null, the p-value must be valid, meaning \(P(p_i\leq t)\leq t\) for every \(t\) between 0 and 1.