Why Dependence Matters
In a multi-arm trial, treatment comparisons often share a control group, outcomes may be correlated, and analyses may use overlapping data. Consequently, the test statistics and their p-values need not be independent. A multiplicity procedure that relies on independence or a particular dependence condition may not retain its stated error rate in these settings.
Bonferroni and Holm offer a useful guarantee: they control the familywise error rate without requiring any particular dependence structure among the tests. “Arbitrary dependence” does not mean that the p-values may be invalid. Each p-value must still be valid for its own null hypothesis, and the family must be specified for the intended scientific question.
The requirement on an individual p-value is marginal validity: for every true \(H_i\) and every \(t\) between 0 and 1, \(P(p_i\le t)\le t\). A continuous uniform p-value under its null satisfies this condition with equality; a discrete or conservative p-value may satisfy it with a strict inequality. The joint distribution of \(p_1,\ldots,p_m\) is otherwise unrestricted for the Bonferroni and Holm guarantees below.
Bonferroni: One Threshold for the Family
The single-step Bonferroni procedure rejects \(H_i\) when \(p_i\le\alpha/m\). Its guarantee follows by bounding the probability of a union by the sum of its probabilities. If \(I_0\) contains \(t\) true nulls, then
The first inequality is the union bound; it does not require independent events. The second uses marginal validity, and the last uses \(t\le m\). This proves strong control, including when only some hypotheses in the family are true. If every null is true, the bound can be conservative: the procedure does not need to spend all of \(\alpha\).
Worked Example: Bonferroni With Four Treatment Comparisons
Suppose a hypothetical trial compares four experimental arms with a control. The family is prespecified to contain those four null hypotheses, and the familywise level is \(\alpha=0.05\). The observed p-values are 0.004, 0.012, 0.018, and 0.20. Assume each p-value is valid under its own null; no assumption about their dependence is made.
The Bonferroni threshold is
Comparing each p-value with 0.0125, the values 0.004 and 0.012 are below the threshold, while 0.018 and 0.20 are above it. Bonferroni therefore rejects the first two corresponding null hypotheses and does not reject the other two. This conclusion has strong FWER control at no more than 0.05 under arbitrary dependence, provided the p-values meet the stated validity condition.
Holm: A Step-Down Procedure
Holm’s procedure orders the p-values from smallest to largest, \(p_{(1)}\le\cdots\le p_{(m)}\), carrying along the hypotheses to which they belong. It compares them sequentially with thresholds that increase as the procedure proceeds. At the first failed comparison, it stops and does not reject that hypothesis or any hypothesis with a larger ordered p-value.
The reason this step-down rule controls FWER is that a false rejection of any true null would require an unusually small p-value among the true nulls. Let \(t=|I_0|\), and consider the first true null in the ordered list. If it is at position \(k\), all \(t\) true nulls occur at positions \(k\) through \(m\), so \(m-k+1\ge t\). For this first true null to be rejected, every earlier comparison must have passed and its own comparison must satisfy
Thus, if Holm rejects any true null, at least one of the \(t\) true-null p-values is at most \(\alpha/t\). Applying the union bound and marginal validity gives
This argument covers every configuration of true and false nulls and uses no joint-distribution assumption. It also explains the step-down logic: the later thresholds become less stringent only after earlier, smaller p-values have passed their own tests.
Worked Example: Holm Gains Rejections Without a Dependence Assumption
Use the same hypothetical four-comparison family and \(\alpha=0.05\), but suppose the ordered p-values are 0.004, 0.012, 0.018, and 0.20. Their dependence is unspecified. The Holm thresholds, in order, are
At the first step, \(0.004\le0.0125\), so reject the first null. At the second, \(0.012\le0.0167\), so reject the second. At the third, \(0.018\le0.025\), so reject the third. At the fourth, \(0.20>0.05\), so stop and do not reject the fourth.
Holm rejects three hypotheses, whereas single-step Bonferroni rejects only the first two for these values. The extra rejection does not depend on an assumed correlation pattern: Holm retains strong FWER control under arbitrary dependence when the marginal p-values are valid.
Contrasting Procedures and Their Assumptions
Bonferroni and Holm are not the only ways to control FWER. The important distinction is whether a procedure’s guarantee depends only on marginal p-value validity or also on the joint behaviour of the tests.
| Procedure or approach | Dependence requirement for its usual guarantee | Practical implication |
|---|---|---|
| Bonferroni | No dependence condition; valid marginal p-values are required. | Simple and robust, but it uses the same stringent threshold for every test. |
| Holm | No dependence condition; valid marginal p-values are required. | Step-down thresholds can allow more rejections than single-step Bonferroni. |
| Hochberg | Its standard guarantee relies on independence or conditions that ensure the relevant Simes inequality. | It can be more powerful, but its guarantee should not be assumed under arbitrary dependence. |
| Joint-distribution methods, such as max-statistic calibration | Require an appropriate model or calibration for the joint null distribution; additional conditions may be needed for strong control. | Can use dependence information, but misspecifying that information can undermine error control. |
Hochberg is a step-up alternative: it checks ordered p-values from the largest downward and may reject several hypotheses after a later ordered value passes its threshold. Under independence, or other conditions establishing the required Simes inequality, this can yield more power than Holm. It is not a general replacement for Holm when dependence is unrestricted. In particular, seeing correlated test statistics does not by itself establish the conditions needed for Hochberg.
Other gains in efficiency can come from valid logical restrictions among hypotheses, as in Shaffer-type procedures, or from procedures that explicitly use a correctly specified joint distribution. These are distinct justifications. Logical constraints concern which truth configurations are possible; joint-distribution methods concern dependence. Neither should be treated as evidence that an assumption-dependent step-up procedure is automatically valid.
Worked Example: A Step-Up Decision Is Not a Dependence Guarantee
Consider four hypothetical ordered p-values 0.014, 0.019, 0.021, and 0.20 at \(\alpha=0.05\). Under the Hochberg step-up rule, the thresholds at ranks 1 through 4 are 0.0125, 0.0167, 0.025, and 0.05. Starting from the largest, the fourth p-value fails because \(0.20>0.05\). The third passes because \(0.021\le0.025\), so the step-up rule rejects the first three hypotheses.
For Holm, the first comparison is \(0.014\le0.0125\), which fails. Holm stops and rejects none. The different decisions illustrate how a step-up rule can be less conservative on these data, but they do not establish that Hochberg controls FWER for this particular dependence structure. If independence or another sufficient condition is not justified, the arbitrary-dependence guarantee provided by Holm remains the relevant one.
Diagnostics, Limitations, and Common Pitfalls
The robustness claim is specific: Bonferroni and Holm protect against dependence among otherwise valid p-values. They do not make an invalid marginal p-value valid. In a multi-stage trial, for example, a p-value that ignores interim monitoring, data-dependent selection, or another feature of the analysis may not have the required null distribution. Multiplicity adjustment cannot repair that defect; the p-value must first account for the design that generated it.
Before applying either procedure, check that the hypotheses in the family match the planned inferential claim, that each p-value corresponds to the stated null, and that the p-values are calibrated under their respective true nulls. Shared controls and overlapping outcomes are reasons not to presume independence, but they are not obstacles to Bonferroni or Holm if marginal validity holds. A procedure may be conservative, especially when many hypotheses are tested or when the tests are strongly dependent, but conservatism does not invalidate its upper bound on FWER.
- Mistake: “The tests look positively correlated, so any step-up procedure is safe.” A qualitative impression of positive correlation is not a proof of the dependence condition required by a procedure. Use Holm for a guarantee under arbitrary dependence unless the alternative’s conditions have been established.
- Mistake: “Holm controls error even if the p-values are not valid.” Its proof uses marginal validity for every true null. If a p-value ignores the trial’s stopping or analysis plan, the proof’s premise may fail.
- Mistake: “Because the union bound can be loose, the FWER guarantee is unreliable.” A loose upper bound may make a method conservative, but if its premises hold, it still guarantees FWER no greater than \(\alpha\).
- Mistake: “Choose the family after looking at the results.” The guarantee concerns the specified family. Changing which hypotheses count after seeing their p-values can change the error-control problem and is not justified by the arbitrary-dependence result.
The practical choice is often straightforward: when the family is fixed and marginal p-values are valid but dependence is unknown or unrestricted, Holm is a step-down option that is at least as permissive as single-step Bonferroni. Use a method with additional dependence assumptions only when those assumptions, and any further conditions for strong control, are defensible.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Four true null hypotheses have valid p-values with arbitrary dependence. What Bonferroni threshold controls FWER at 0.05?
- In Holm’s procedure for \(m=5\), what is the threshold at the third ordered p-value?
- Why does a valid marginal p-value not require the p-values from different treatment comparisons to be independent?
- A researcher says that Holm will correct a p-value calculated without accounting for an interim stopping rule. What is wrong with this reasoning?
- At \(\alpha=0.05\), four ordered p-values are 0.006, 0.014, 0.030, and 0.40. How many does Holm reject?
Solutions. (1) The threshold is \(0.05/4=0.0125\). (2) Holm’s third threshold is \(0.05/(5-3+1)=0.05/3\approx0.0167\). (3) Marginal validity bounds each true-null event \(P(p_i\le t)\); the union-bound proof sums these bounds and does not use their joint distribution. (4) Holm requires each true-null p-value to be valid. If the interim stopping rule is ignored and invalidates that p-value’s null calibration, multiplicity adjustment cannot restore validity. (5) The thresholds are 0.0125, 0.0167, 0.025, and 0.05. The first passes because \(0.006\le0.0125\); the second passes because \(0.014\le0.0167\); the third fails because \(0.030>0.025\). Holm stops there and rejects exactly two hypotheses.