Holm-Bonferroni Step-Down Procedure

Introduction

When a clinical trial or other statistical study evaluates several hypotheses, the probability of obtaining at least one apparently significant result can be larger than the nominal significance level.

For example, if five independent hypotheses are each tested at the conventional 0.05 level, the probability of at least one false positive can be substantially greater than 5%. This is the fundamental multiplicity problem.

The Holm-Bonferroni procedure, also called the Holm step-down procedure, provides a simple way to control the family-wise error rate (FWER) while generally being less conservative than applying the ordinary Bonferroni correction to every hypothesis at the same threshold.

Key idea: Holm does not use one common adjusted significance threshold for every hypothesis. Instead, it orders the p-values from smallest to largest and applies progressively less stringent thresholds as it moves through the ordered hypotheses.

The Multiple Testing Problem

Suppose a study evaluates \(m\) null hypotheses:

H_1,H_2,\ldots,H_m

and produces corresponding p-values:

p_1,p_2,\ldots,p_m.

If each hypothesis is tested independently at level \(\alpha=0.05\), the probability of making at least one Type I error can exceed 0.05.

The family-wise error rate is:

\operatorname{FWER} = P(\text{at least one false rejection in the family}).

A multiplicity adjustment attempts to ensure:

\operatorname{FWER}\le\alpha.

Clinical interpretation: If a confirmatory trial has several hypotheses that collectively form a statistical testing family, controlling FWER at 5% means that the probability of making one or more false positive efficacy claims within that family is controlled at 5% or less.

The Ordinary Bonferroni Correction

The simplest FWER adjustment is the Bonferroni correction. For \(m\) hypotheses, each individual hypothesis is tested at:

\alpha_{\text{Bonferroni}}=\frac{\alpha}{m}.

Equivalently, a p-value can be multiplied by \(m\):

p_{i,\text{Bonferroni}} = \min(mp_i,1).

The Bonferroni method is attractive because it is simple and makes very few assumptions about the dependence among tests.

However, it can be conservative, particularly when many hypotheses are being tested and some p-values are extremely small.

What Makes Holm Different?

Holm's procedure retains the FWER control of Bonferroni but uses the information contained in the ordering of the p-values.

The p-values are first sorted:

p_{(1)}\le p_{(2)}\le\cdots\le p_{(m)}.

The smallest p-value is compared with the most stringent threshold:

\frac{\alpha}{m}.

The second-smallest p-value is compared with:

\frac{\alpha}{m-1}.

The process continues until:

\frac{\alpha}{m-j+1}

is used for the \(j\)-th ordered p-value.

Key difference: Bonferroni uses \(\alpha/m\) for every test. Holm begins with \(\alpha/m\), but after a successful rejection the next threshold becomes \(\alpha/(m-1)\), then \(\alpha/(m-2)\), and so on.

The Holm Step-Down Algorithm

Suppose there are \(m\) hypotheses and the desired family-wise error rate is \(\alpha\). The Holm procedure can be implemented in the following sequence.

Collect the p-values. Obtain one p-value for each hypothesis in the prespecified testing family.

Sort the p-values. Order them from smallest to largest: \(p_{(1)}\le\cdots\le p_{(m)}\).

Test the smallest p-value. Compare \(p_{(1)}\) with \(\alpha/m\).

Continue only after rejection. If \(p_{(1)}\le\alpha/m\), reject \(H_{(1)}\) and move to the next p-value.

Increase the threshold. For \(p_{(j)}\), compare it with \(\alpha/(m-j+1)\).

Stop at the first failure. If \(p_{(j)}>\alpha/(m-j+1)\), do not reject \(H_{(j)}\) or any subsequent hypothesis.

The Mathematical Rule

For the \(j\)-th ordered p-value, Holm compares:

p_{(j)} \quad\text{with}\quad \frac{\alpha}{m-j+1}.

The decision rule is therefore:

p_{(j)} \le \frac{\alpha}{m-j+1} \quad\Rightarrow\quad \text{reject }H_{(j)}.

If instead:

p_{(j)} > \frac{\alpha}{m-j+1},

the procedure stops. All remaining hypotheses are not rejected.

The step-down stopping rule is essential. Holm is not a procedure in which each p-value is evaluated independently against its own threshold. Once an ordered hypothesis fails its Holm comparison, all larger p-values are also left unrejected.

A Complete Worked Example

Suppose a clinical trial produces five p-values from a prespecified family of hypothesis tests:

Hypothesis	Raw p-value
\(H_1\)	0.003
\(H_2\)	0.012
\(H_3\)	0.018
\(H_4\)	0.041
\(H_5\)	0.200

We want to control the family-wise error rate at:

\alpha=0.05.

There are:

\[ m=5 \]

hypotheses.

Step 1: Sort the P-Values

The p-values are already in ascending order:

0.003 \le 0.012 \le 0.018 \le 0.041 \le 0.200.

The ordered p-values are therefore:

Rank \(j\)	Ordered p-value	Original Hypothesis
1	0.003	\(H_1\)
2	0.012	\(H_2\)
3	0.018	\(H_3\)
4	0.041	\(H_4\)
5	0.200	\(H_5\)

Step 2: Test the Smallest P-Value

For the first p-value, the Holm threshold is:

\frac{0.05}{5}=0.010.

We compare:

0.003\le0.010.

Therefore, \(H_1\) is rejected.

First decision: The smallest p-value is sufficiently small to survive the most stringent Holm threshold.

Step 3: Test the Second P-Value

Because the first hypothesis was rejected, we continue. For the second ordered p-value, the denominator decreases from 5 to 4:

\frac{0.05}{4}=0.0125.

Now compare:

0.012\le0.0125.

This is also significant under the Holm procedure. Therefore, \(H_2\) is rejected.

Step 4: Test the Third P-Value

For the third ordered p-value, the threshold is:

\frac{0.05}{3}=0.01667.

The comparison is:

\[ 0.018>0.01667. \]

The third p-value fails its Holm threshold.

This is where the step-down procedure stops. Because the third ordered p-value fails, \(H_3\), \(H_4\), and \(H_5\) are not rejected by the Holm procedure.

Complete Step-Down Table

Rank	p-value	Holm Threshold	Comparison	Decision
1	0.003	0.01000	0.003 ≤ 0.01000	Reject
2	0.012	0.01250	0.012 ≤ 0.01250	Reject
3	0.018	0.01667	0.018 > 0.01667	Stop
4	0.041	0.02500	Not evaluated after failure	Do not reject
5	0.200	0.05000	Not evaluated after failure	Do not reject

The final conclusion is therefore:

\boxed{\text{Reject }H_1\text{ and }H_2;\quad \text{do not reject }H_3,H_4,H_5.}

Holm-Adjusted P-Values

Instead of reporting only the sequential significance decisions, one can report Holm-adjusted p-values.

For the ordered p-values, the basic Holm-adjusted quantity is:

(m-j+1)p_{(j)}.

To ensure that adjusted p-values preserve the required monotonicity, the adjusted values are obtained using the cumulative maximum:

p_{(j),\text{Holm}} = \max_{k\le j} \left\{ (m-k+1)p_{(k)} \right\},

with the adjusted p-value truncated at 1 if necessary.

Calculating the Example

For the first p-value:

\[ 5(0.003)=0.015. \]

For the second:

\[ 4(0.012)=0.048. \]

For the third:

\[ 3(0.018)=0.054. \]

For the fourth:

\[ 2(0.041)=0.082. \]

For the fifth:

\[ 1(0.200)=0.200. \]

These values are already nondecreasing, so the cumulative maximum does not change them.

Hypothesis	Raw p-value	Holm-adjusted p-value	Adjusted Decision at 0.05
\(H_1\)	0.003	0.015	Reject
\(H_2\)	0.012	0.048	Reject
\(H_3\)	0.018	0.054	Do not reject
\(H_4\)	0.041	0.082	Do not reject
\(H_5\)	0.200	0.200	Do not reject

Equivalent interpretation: A hypothesis is rejected at family-wise level \(\alpha\) if its Holm-adjusted p-value is at most \(\alpha\). Thus, the adjusted-p-value approach and the step-down threshold approach produce the same rejection set.

Why Holm Is More Powerful Than Bonferroni

The ordinary Bonferroni procedure would test every hypothesis against:

\frac{0.05}{5}=0.010.

Under ordinary Bonferroni, the p-values 0.003 would be significant, but 0.012 would not be.

Holm, however, rejects the second p-value because after the smallest hypothesis has been rejected, the second threshold becomes:

\frac{0.05}{4}=0.0125.

Thus:

0.012\le0.0125.

Holm can therefore reject hypotheses that ordinary Bonferroni would leave unrejected.

Procedure	Threshold for p-value 1	Threshold for p-value 2	Result in Example
Bonferroni	0.0100	0.0100	Reject only \(H_1\)
Holm	0.0100	0.0125	Reject \(H_1\) and \(H_2\)

Important: Holm is not less conservative because it gives up FWER control. It is more efficient because it uses a sequential procedure that exploits the ordering of the observed p-values.

Bonferroni vs. Holm

Feature	Bonferroni	Holm-Bonferroni
Controls FWER	Yes	Yes
Requires independent tests	No	No
Sorts p-values	No	Yes
Uses step-down procedure	No	Yes
Thresholds change during testing	No	Yes
Generally more powerful	No	Yes
Simple adjusted p-values	Yes	Yes

Holm vs. Hochberg

Holm and Hochberg are both sequential multiplicity procedures, but they operate in opposite directions.

Holm is step-down: it starts with the smallest p-value and moves upward.

Hochberg is step-up: it starts with the largest p-value and moves downward.

Feature	Holm	Hochberg
Direction	Step-down	Step-up
Starts with	Smallest p-value	Largest p-value
FWER control	Yes	Yes under appropriate dependence conditions
General dependence robustness	Strong	More restrictive

Holm is often an attractive default when robust FWER control is desired without relying on stronger assumptions about the dependence structure among tests.

Holm in Clinical Trials

Multiplicity is common in clinical trials. A study may contain multiple hypotheses involving:

multiple doses;
multiple endpoints;
multiple treatment comparisons;
different populations;
different time points;
hierarchical testing families; or
other prespecified confirmatory comparisons.

The first question is not simply whether Holm can be calculated. The more important question is whether the hypotheses belong to the same multiplicity family.

Planning principle: The testing family and multiplicity strategy should be defined prospectively. Do not decide after seeing the results which hypotheses should be grouped together in order to obtain a favorable conclusion.

A Simple Clinical Example

Suppose a Phase III trial evaluates an experimental treatment against placebo on three confirmatory endpoints:

Endpoint	Raw p-value
Primary endpoint	0.008
Key secondary endpoint 1	0.021
Key secondary endpoint 2	0.030

Suppose the family-wise error rate must be controlled at 5%. There are \(m=3\) hypotheses.

The Holm thresholds are:

\frac{0.05}{3}=0.01667, \qquad \frac{0.05}{2}=0.025, \qquad \frac{0.05}{1}=0.05.

The first comparison is:

\[ 0.008<0.01667, \]

so the primary endpoint is rejected.

The second comparison is:

\[ 0.021<0.025, \]

so the second endpoint is also rejected.

The third comparison is:

\[ 0.030<0.05. \]

Therefore, all three hypotheses are rejected by this particular Holm analysis.

Do not confuse statistical rejection with clinical importance. A Holm-adjusted rejection establishes statistical evidence within the specified multiplicity framework. Clinical interpretation still requires consideration of effect size, confidence intervals, endpoint relevance, and the prespecified estimand.

Handling Ties in P-Values

Occasionally two or more hypotheses have identical p-values. For example:

0.004,\;0.010,\;0.010,\;0.032.

The Holm procedure can still be applied after sorting. The hypotheses associated with tied p-values receive the corresponding ordered ranks according to the implementation.

The key requirement is that the resulting rejection decisions and adjusted p-values be calculated consistently.

Why the Procedure Must Stop After the First Failure

Suppose:

p_{(j)} > \frac{\alpha}{m-j+1}.

If the procedure continued independently to later p-values, it could potentially reject a larger p-value after failing to reject a smaller one. That would violate the intended step-down structure.

Instead, Holm imposes the rule:

\text{first failure} \quad\Rightarrow\quad \text{stop and reject no subsequent hypotheses}.

This produces a rejection set that is ordered and coherent with the step-down procedure.

Common Mistakes

1. Dividing Every P-Value by the Same Number

That is essentially the ordinary Bonferroni approach, not the Holm step-down procedure.

2. Using the Original Hypothesis Order

Holm operates on the ordered p-values, not the order in which the hypotheses happen to appear in a table or protocol.

3. Continuing After a Failure

Once an ordered p-value fails its Holm threshold, subsequent hypotheses are not rejected.

4. Forgetting to Restore the Original Hypothesis Labels

The p-values are sorted for the calculation, but the final results must be mapped back to the corresponding original hypotheses or endpoints.

5. Treating Holm as a Post-Hoc License to Test Everything

Holm can control FWER for a defined family of hypotheses, but it does not remove the need to define an appropriate testing family and analysis strategy.

6. Reporting Only Raw P-Values

When multiplicity adjustment is part of the confirmatory analysis, the statistical report should make clear which multiplicity procedure was used and which conclusions are based on adjusted inference.

Implementation in R

In R, the base p.adjust() function provides Holm adjustment directly.

p <- c(0.003, 0.012, 0.018, 0.041, 0.200)

p.adjust(p, method = "holm")

For the worked example, the resulting adjusted p-values are:

0.015 0.048 0.054 0.082 0.200

A decision at the 5% family-wise level can then be obtained with:

p.adjust(p, method = "holm") <= 0.05

which returns:

TRUE TRUE FALSE FALSE FALSE

Implementation in SAS

In SAS, multiplicity adjustments can be requested through procedures that support multiple-comparison adjustments. For example, in an appropriate model-based analysis, the ADJUST=HOLM option can be used where supported by the procedure.

proc glm data=analysis;
    class treatment;
    model endpoint = treatment;
    lsmeans treatment / pdiff=all adjust=holm;
run;

The exact SAS syntax depends on the statistical procedure and estimand being analyzed. The multiplicity method should be selected in the context of the specific analysis rather than mechanically added to every model.

Programming principle: Verify that the procedure actually implements the intended Holm adjustment and that the multiplicity family represented by the software matches the family specified in the statistical analysis plan.

Holm-Adjusted P-Values vs. Sequential Testing

There are two equivalent ways to communicate the Holm procedure:

compare ordered p-values sequentially with their Holm thresholds; or
calculate Holm-adjusted p-values and compare them with \(\alpha\).

These approaches lead to the same rejection decisions when implemented correctly.

Approach	Main Calculation	Typical Reporting Use
Step-down thresholds	\(p_{(j)}\le\alpha/(m-j+1)\)	Explaining the algorithm
Adjusted p-values	\(\max_{k\le j}(m-k+1)p_{(k)}\)	Tables and statistical reports

Practical Holm Workflow

Define the family of hypotheses for which FWER control is required.

Specify the desired family-wise significance level, commonly \(\alpha=0.05\).

Calculate the individual unadjusted p-values using the prespecified statistical analyses.

Sort the p-values from smallest to largest while retaining their hypothesis labels.

Compare the smallest p-value with \(\alpha/m\).

After each rejection, move to the next p-value and use \(\alpha/(m-j+1)\).

Stop at the first p-value that fails its threshold.

Map the decisions back to the original hypotheses or endpoints.

If desired, calculate and report Holm-adjusted p-values.

Document the multiplicity family, adjustment method, alpha level, and resulting confirmatory conclusions.

What Should Be Reported in a Statistical Analysis Plan?

When Holm is used to control multiplicity in a clinical trial, the statistical analysis plan should clearly describe the testing strategy.

A useful statistical section should specify:

the hypotheses included in the multiplicity family;
the endpoint or estimand corresponding to each hypothesis;
the order of any prespecified hierarchical testing strategy, if applicable;
the family-wise significance level;
the multiplicity adjustment method;
the statistical model or test producing each raw p-value;
the rule for calculating adjusted p-values, if they will be reported;
the treatment of missing or unevaluable data;
the population used for the confirmatory analysis; and
the method used to report and interpret multiplicity-adjusted conclusions.

Important distinction: Holm is a multiplicity adjustment procedure. It does not determine how the underlying endpoint should be analyzed. The model, estimand, contrast, confidence interval, and raw p-value calculation must still be appropriately specified.

Key Takeaways

Concept	Takeaway
Purpose	Control the family-wise error rate when testing multiple hypotheses.
First step	Sort p-values from smallest to largest.
Threshold	The \(j\)-th ordered p-value is compared with \(\alpha/(m-j+1)\).
Direction	Holm is a step-down procedure.
Stopping rule	Stop at the first ordered p-value that fails its threshold.
Power	Generally more powerful than ordinary Bonferroni while retaining FWER control.
Adjusted p-values	Can be reported instead of, or alongside, sequential threshold comparisons.
Clinical use	Useful for multiplicity families in confirmatory clinical-trial analyses.

Final Perspective

The Holm-Bonferroni procedure is one of the most useful general-purpose tools for controlling multiplicity.

Its logic is straightforward: sort the p-values, test the smallest one first, progressively relax the threshold after successful rejections, and stop at the first failure.

Compared with ordinary Bonferroni, Holm can recover additional statistically significant findings without sacrificing family-wise error control.

For clinical trials, however, the arithmetic is only one part of the problem. The hypotheses included in the family, the estimands, endpoint definitions, testing strategy, and interpretation of the resulting evidence should all be specified prospectively.

Bottom line: For \(m\) hypotheses, Holm compares the ordered p-values sequentially against \(\alpha/m,\alpha/(m-1),\ldots,\alpha\). It controls the family-wise error rate and is never less powerful than ordinary Bonferroni for the same family and significance level.