Why Compare the Procedures on the Same Three Tests?
In a multi-arm trial, each of three treatments may be compared with a common control on the same primary outcome. Testing each comparison at the nominal level \(\alpha\) without accounting for multiplicity can make the probability of one or more false rejections across the family exceed \(\alpha\). Here we work through the same family of three hypotheses using Bonferroni and Holm, so that the role of each procedure’s rule is visible.
The numerical examples are hypothetical. Assume that the three p-values come from valid tests chosen in advance for the three treatment-versus-control contrasts. For illustration, take the familywise significance level to be \(\alpha=0.05\). The family is fixed as all three comparisons, regardless of what their results turn out to be. Neither procedure requires the three tests to be independent.
A valid p-value for a true null is super-uniform: for every \(u\) between 0 and 1, \(P(p\leq u)\leq u\). This condition, rather than independence, is what is needed for the FWER guarantees described here. The individual tests must also correspond to the prespecified hypotheses and analysis choices: for example, the endpoint, contrast, and one- or two-sided alternative should not be selected after inspecting the results.
Worked Example: All Three Holm Comparisons Pass
Worked Example: All Three Holm Comparisons Pass
Suppose three hypothetical p-values for treatments A, B, and C versus the control are \(0.006\), \(0.019\), and \(0.044\), respectively. The three comparisons form the family, and the prespecified familywise level is \(0.05\).
Bonferroni. The per-comparison threshold is
Treatment A has \(0.006\leq0.01667\), so its null hypothesis is rejected. Treatment B has \(0.019>0.01667\), and treatment C has \(0.044>0.01667\), so neither of those null hypotheses is rejected. The Bonferroni rejection set is therefore \(\{A\}\).
Holm. The ordered p-values are \(p_{(1)}=0.006\) for A, \(p_{(2)}=0.019\) for B, and \(p_{(3)}=0.044\) for C. Compare them sequentially:
All three comparisons pass, so Holm rejects all three null hypotheses. In context, at the 0.05 FWER level, both procedures reject the treatment A comparison, while Holm also rejects the comparisons for treatments B and C. These are decisions about the evidence against the corresponding null hypotheses, not measures of the size or clinical importance of treatment effects.
Holm’s Step-Down Rule and Why It Controls FWER
The essential feature of Holm’s procedure is that later, less stringent thresholds are available only if all earlier comparisons have passed. For \(m\) hypotheses, write the ordered p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\). At rank \(i\), Holm compares \(p_{(i)}\) with \(\alpha/(m-i+1)\). If it passes, the null hypothesis at that rank is rejected and the procedure moves on. At the first failure, the procedure stops; it does not skip that hypothesis and test later ones.
For three hypotheses, the successive thresholds are \(\alpha/3\), \(\alpha/2\), and \(\alpha\). If the hypotheses at ranks 1 and 2 pass, the hypothesis at rank 3 is tested at \(\alpha\). If the first comparison fails, none is rejected. This stopping rule is part of the procedure, not an optional convention.
The reason Holm controls FWER under arbitrary dependence can be seen by considering the true null hypotheses. Let \(t\) be their number, and suppose at least one true null is rejected. In the ordered list, let the first true null occur at rank \(j\). For it to be rejected, all earlier comparisons must have passed, and its p-value must satisfy \(p_{(j)}\leq\alpha/(m-j+1)\). Since the \(t\) true nulls all occur at rank \(j\) or later, \(t\leq m-j+1\), and hence
Therefore, if any true null is rejected, at least one of the \(t\) true-null p-values must be at most \(\alpha/t\). By the union bound and the validity of each true-null p-value,
This establishes strong FWER control: the bound holds for any configuration of true and false null hypotheses, and it does not rely on independence. Bonferroni’s guarantee follows directly in the same way: if any true null is rejected, one of the true-null p-values must be at most \(\alpha/m\), so the union bound gives an error probability no greater than \(m(\alpha/m)=\alpha\).
Worked Example: Holm Rejects Two, Bonferroni Rejects One
Worked Example: Holm Rejects Two, Bonferroni Rejects One
Now suppose the hypothetical p-values are \(p_A=0.024\), \(p_B=0.052\), and \(p_C=0.011\), with the same three-hypothesis family and \(\alpha=0.05\).
Bonferroni. The threshold remains \(0.05/3=0.01667\). Only \(p_C=0.011\) is at or below this threshold; \(p_A=0.024\) and \(p_B=0.052\) are above it. The rejection set is \(\{C\}\).
Holm. First order the p-values: \(p_{(1)}=0.011\) for C, \(p_{(2)}=0.024\) for A, and \(p_{(3)}=0.052\) for B. The first comparison passes because \(0.011\leq0.01667\). The second passes because \(0.024\leq0.025\). The third fails because \(0.052>0.05\), so Holm stops. Its rejection set is \(\{C,A\}\), and B is not rejected.
The two procedures therefore agree on C but differ on A: Holm rejects A after C passes the first step, whereas Bonferroni compares A with the same \(\alpha/3\) threshold as every other comparison. The rejection decisions apply to the named treatment comparisons; reporting only “the first two were rejected” would be incomplete unless the treatment labels were also given.
Worked Example: The First Holm Failure Stops the Procedure
Worked Example: The First Holm Failure Stops the Procedure
Suppose the hypothetical p-values are \(p_A=0.018\), \(p_B=0.049\), and \(p_C=0.017\). The family and \(\alpha=0.05\) are unchanged.
Bonferroni. Each p-value is compared with \(0.01667\). Since \(0.018>0.01667\), \(0.049>0.01667\), and \(0.017>0.01667\), none is rejected.
Holm. The ordered values are \(p_{(1)}=0.017\) for C, \(p_{(2)}=0.018\) for A, and \(p_{(3)}=0.049\) for B. The first comparison fails:
Holm stops immediately, so none of the hypotheses is rejected. It would be incorrect to continue and compare \(0.018\) with \(0.025\) or \(0.049\) with \(0.05\): those later thresholds are reached only when every earlier comparison has passed. The fact that these later p-values would be below their corresponding thresholds does not change the procedure’s rejection set.
Reading and Reporting the Results
A clear report names the family, the adjustment, and the decisions. For example: “For the prespecified family of three treatment-versus-control tests at FWER level 0.05, Holm rejected the comparisons for treatments C and A but not B; Bonferroni rejected only the comparison for C.” If the scientific question also concerns effect size, report the treatment-effect estimates and an appropriate interval separately. A multiplicity-adjusted rejection decision alone does not give the estimated difference, its units, or its clinical relevance.
Bonferroni and Holm both control FWER at level \(\alpha\) when their assumptions hold. Holm is uniformly at least as powerful as Bonferroni in the sense that it never rejects fewer hypotheses for the same family and \(\alpha\). To see this, order the p-values. Bonferroni rejects the values at or below \(\alpha/m\). If a value at rank \(i\) satisfies this condition, then it also satisfies the Holm threshold \(\alpha/(m-i+1)\), which is at least \(\alpha/m\); every earlier ordered p-value is no larger and also passes its Holm threshold. Thus Holm reaches and rejects that rank. The procedures can still yield the same rejection set, as in the first example.
| Feature | Bonferroni | Holm |
|---|---|---|
| Comparison rule for three tests | Compare every p-value with \(\alpha/3\) | Order p-values and compare successively with \(\alpha/3,\alpha/2,\alpha\) |
| Stopping rule | No step-down stopping rule | Stop at the first failed comparison |
| FWER control | Yes, under valid individual p-values | Yes, under valid individual p-values |
| Dependence requirement | No independence required | No independence required |
Common Mistakes and Practical Checks
- Testing every arm at 0.05. This ignores the three-hypothesis family. Apply the specified multiplicity procedure to the family rather than treating each comparison as a separate, unadjusted primary test.
- Using Holm thresholds but not its stopping rule. If a comparison fails, stop; do not test a later hypothesis at its more permissive threshold.
- Changing the family after seeing the results. Dropping an unattractive comparison changes the planned error-control problem. Keep all prespecified hypotheses in the family when making the stated decisions.
- Confusing order with treatment identity. Sort p-values to apply Holm, but translate every rank back to its treatment label when reporting decisions.
- Rounding before a borderline decision. Use the unrounded p-values and thresholds when available. For instance, a displayed value of 0.0167 could lie slightly above or below the exact threshold \(0.05/3\).
Before reporting a result, check that there are exactly three prespecified comparisons, that all p-values come from the intended valid tests, and that the same \(\alpha\) and sidedness were used throughout. If a p-value is invalid because of a misspecified test or a post hoc choice of analysis, applying an adjustment to it does not repair that problem.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions.
- For \(m=3\) and \(\alpha=0.05\), what are the three Holm thresholds?
- With ordered p-values \(0.010, 0.024, 0.048\), which hypotheses does Holm reject? State the rank-based calculations.
- With ordered p-values \(0.018, 0.019, 0.030\), can Holm reject the second hypothesis? Explain.
- Why does Holm not require independence among the three tests?
Solutions. (1) The thresholds are \(0.05/3=0.01667\), \(0.05/2=0.025\), and \(0.05/1=0.05\). (2) All three pass: \(0.010\leq0.01667\), \(0.024\leq0.025\), and \(0.048\leq0.05\); Holm rejects the hypotheses at ranks 1, 2, and 3. (3) No. The first value fails because \(0.018>0.01667\), so the procedure stops and rejects none, even though \(0.019\leq0.025\). (4) The FWER guarantee follows from the union bound applied to valid true-null p-values; that bound does not require independence.