Why a Step-Down Rule Is Useful
A multi-arm trial may compare several treatments with a shared control. If each comparison is tested at the same unadjusted significance level, the chance of making at least one false rejection across the set of comparisons can exceed that level. The family of hypotheses must therefore be handled together. Holm’s step-down procedure gives a simple way to make decisions across that family: order the p-values, compare them with a sequence of thresholds, and stop at the first failure.
The key feature is that the threshold becomes less stringent as the procedure moves from the smallest p-value to the largest. Unlike a single Bonferroni cutoff, Holm’s rule can therefore reject some hypotheses whose p-values are larger than that single cutoff, while still requiring the comparisons to be assessed as one prespecified family.
Set Up the Family and the Test Results
Let the prespecified family contain \(m\) null hypotheses, \(H_1,\ldots,H_m\), such as the null hypotheses that each of \(m\) experimental treatments has no effect relative to a common control. For each \(H_j\), calculate a p-value \(p_j\) from a test appropriate to that hypothesis and the trial’s analysis plan. The procedure acts on these p-values; it does not specify how to estimate treatment effects or construct the underlying tests.
The family should be defined before looking at the results. For example, if the primary family consists of three treatment-versus-control comparisons, then \(m=3\); it is not appropriate to omit an inconvenient comparison after seeing its p-value. In a multi-arm trial, estimates that share a control group may be statistically dependent. Independence is not needed to carry out Holm’s ordering and threshold comparisons.
The Ordered Thresholds and Stopping Rule
Sort the p-values from smallest to largest and denote them by \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). The notation \(H_{(i)}\) means the hypothesis associated with the p-value in position \(i\), not necessarily the hypothesis with the original label \(i\). At position \(i\), compare \(p_{(i)}\) with \(\alpha/(m-i+1)\).
Start at \(i=1\). If the p-value is at or below its threshold, reject its null hypothesis and proceed to the next position. If it is above its threshold, stop: do not reject that hypothesis or any hypothesis later in the ordering. Thus, the rejected hypotheses are exactly the hypotheses occupying the initial sequence of positions that passed every comparison up to the stopping point. Equality counts as passing.
The denominator shrinks by one at each step, so the threshold increases as the procedure proceeds. Passing a comparison does not mean that all remaining hypotheses will be rejected: every remaining p-value must pass its own threshold, and one failure ends the process.
Worked Example: Three Treatment Comparisons All Pass
Suppose a hypothetical trial has three prespecified treatment-versus-control hypotheses, \(H_A,H_B,H_C\), and uses \(\alpha=0.05\). Assume the p-values come from valid tests specified for the trial. The hypothetical observed p-values are \(p_A=0.018\), \(p_B=0.004\), and \(p_C=0.041\). Order them before applying the thresholds.
The sorted sequence is \(p_{(1)}=0.004\) for \(H_B\), \(p_{(2)}=0.018\) for \(H_A\), and \(p_{(3)}=0.041\) for \(H_C\). Since \(m=3\), the thresholds are \(0.05/3=0.0167\), \(0.05/2=0.0250\), and \(0.05/1=0.0500\).
At position 1, \(0.004\leq0.0167\), so reject \(H_B\) and continue. At position 2, \(0.018\leq0.0250\), so reject \(H_A\) and continue. At position 3, \(0.041\leq0.0500\), so reject \(H_C\). All three null hypotheses are rejected by Holm’s procedure at familywise level \(0.05\). In context, the procedure identifies evidence against each of the three treatment-versus-control null hypotheses; it does not establish that the treatments have equal or different effect sizes.
Worked Example: Stop at the First Failure
Consider four hypothetical treatment comparisons, \(H_1,\ldots,H_4\), in a prespecified family, with \(\alpha=0.05\). Suppose their valid p-values, after ordering, are \(0.006\) for \(H_3\), \(0.019\) for \(H_1\), \(0.022\) for \(H_4\), and \(0.031\) for \(H_2\). The hypotheses are shown in ordered position, so their labels need not match their positions.
Here \(m=4\). The thresholds in order are \(0.05/4=0.0125\), \(0.05/3=0.0167\), \(0.05/2=0.0250\), and \(0.05/1=0.0500\).
The first comparison passes because \(0.006\leq0.0125\), so reject \(H_3\) and continue. The second fails because \(0.019>0.0167\). Stop at this point: reject \(H_3\), but do not reject \(H_1\), \(H_4\), or \(H_2\) under Holm’s procedure. In particular, \(0.022\leq0.0250\) does not allow \(H_4\) to be rejected after the earlier failure. Holm’s rule is sequential, not a set of independent comparisons made in any order.
Worked Example: Tied P-Values and Position-Specific Thresholds
Suppose a hypothetical three-arm comparison produces three p-values \(0.008\), \(0.020\), and \(0.020\), for hypotheses \(H_C\), \(H_A\), and \(H_B\), respectively, with \(\alpha=0.05\). Assume all three p-values are valid for their corresponding null hypotheses.
For \(m=3\), the ordered thresholds are \(0.0167\), \(0.0250\), and \(0.0500\). At the first position, \(0.008\leq0.0167\), so reject \(H_C\). At the second position, \(0.020\leq0.0250\), so reject one of the tied hypotheses and continue. At the third position, \(0.020\leq0.0500\), so reject the other tied hypothesis. Both hypotheses with p-value \(0.020\) are rejected. Because the values are tied, either may be listed first; both comparisons pass at their respective positions, so the final decisions do not depend on how the tie is listed here.
Why the Family Definition Matters
Holm’s thresholds depend on \(m\), the number of hypotheses in the family. If the planned family contains \(m=4\) hypotheses, the first threshold at \(\alpha=0.05\) is \(0.0125\). If one hypothesis is improperly removed after results are known, the first threshold would instead be \(0.0167\), and later thresholds would also change. Such post-result changes alter the procedure rather than merely relabeling it.
A useful operational record is a table that includes the original hypothesis label, its raw p-value, its ordered position, its Holm threshold, and the decision. This makes it possible to verify that the p-values were sorted correctly and that no comparison was tested after the first failure.
| Position | Threshold at \(\alpha\) | Decision rule |
|---|---|---|
| 1 | \(\alpha/m\) | Pass to position 2 only if \(p_{(1)}\leq\alpha/m\) |
| 2 | \(\alpha/(m-1)\) | Pass to position 3 only if position 1 passed and \(p_{(2)}\leq\alpha/(m-1)\) |
| \(i\) | \(\alpha/(m-i+1)\) | Continue only if all earlier positions and position \(i\) passed |
| \(m\) | \(\alpha\) | Reject the final hypothesis only if every earlier position passed and \(p_{(m)}\leq\alpha\) |
Relationship to a Single Bonferroni Cutoff
The ordinary Bonferroni procedure rejects a hypothesis only when its p-value is at most \(\alpha/m\). Holm starts with that same threshold for the smallest p-value, but then uses larger thresholds for later ordered p-values if all earlier comparisons passed. For the first worked example, the Bonferroni cutoff is \(0.05/3=0.0167\): it would reject only the hypothesis with p-value \(0.004\). Holm also rejects the hypotheses with p-values \(0.018\) and \(0.041\), because they pass the later step-specific thresholds.
Holm’s procedure is a step-down method: it begins with the most statistically significant result and can stop early. It should not be confused with applying a separate Bonferroni test to each arm using a denominator that is changed after seeing which comparisons look promising. The full ordering and stop rule are part of the procedure.
Assumptions, Diagnostics, and Common Errors
The calculation is straightforward, but its interpretation depends on a sound analysis setup. The family and \(\alpha\) should be specified, each p-value should be valid for its null hypothesis, and all hypotheses in the family should be included in the ordering. Holm’s procedure is a standard method for controlling the familywise error rate, the probability of at least one false rejection in the family, at level \(\alpha\) when the individual p-values are valid. This property does not require independent tests; its justification is distinct from the step-by-step calculation.
- Continuing after a failure. A common error is to stop at \(p_{(i)}\) that exceeds its threshold, then reject a later hypothesis because its p-value is below a later, larger threshold. The correction is to stop at the first failure and make no later rejections.
- Using the wrong denominator. The denominator at position \(i\) is \(m-i+1\), not always \(m\), and not the number of hypotheses remaining after any investigator-selected exclusions.
- Forgetting to sort. The thresholds apply to ordered p-values. Applying \(\alpha/m,\alpha/(m-1),\ldots\) to treatment labels in their original listing is not Holm’s procedure.
- Changing the family after seeing results. Removing a treatment comparison changes both the family and the thresholds. Use the prespecified family for the primary analysis.
- Treating a p-value as an effect measure. A small p-value does not say how large or clinically important a treatment effect is. Report effect estimates and suitable intervals alongside multiplicity-adjusted decisions.
If the reported decisions seem inconsistent, check the family size, ordering, arithmetic, and stopping point in that order. Also verify that the input p-values correspond to the prespecified tests, including the intended endpoint, direction, and analysis population. Holm’s procedure cannot repair invalid or selectively chosen p-values.
Check Your Understanding
For each problem, use \(\alpha=0.05\) unless another value is specified. The solutions follow the questions.
- A family has three p-values, \(0.010\), \(0.021\), and \(0.049\). Apply Holm’s procedure and state how many hypotheses are rejected.
- For four ordered p-values \(0.004\), \(0.018\), \(0.020\), and \(0.040\), identify the first failure and state the rejected positions.
- Why must the p-values be sorted before the thresholds are applied?
- In a family of five hypotheses, what is the threshold at the third ordered position?
Solutions. (1) The thresholds are \(0.0167\), \(0.0250\), and \(0.0500\). Each p-value passes its threshold, so all three hypotheses are rejected. (2) The thresholds are \(0.0125\), \(0.0167\), \(0.0250\), and \(0.0500\). The first value passes; the second fails because \(0.018>0.0167\). Stop there and reject only the first ordered hypothesis. (3) The sequence of thresholds is assigned to the smallest through largest p-values; applying it to unsorted treatment labels changes the test and its decisions. (4) At position 3, the threshold is \(\alpha/(5-3+1)=0.05/3=0.0167\).