Holm's Step-Down Procedure

Why Use an Ordered Procedure?

In genomic analyses, a testing family may contain many hypotheses, such as tests of association for multiple variants. Controlling the familywise error rate (FWER) means limiting the probability of making at least one false rejection in that family. A single common Bonferroni threshold protects this error rate, but it treats every rank in the ordered list of p-values alike. Holm’s procedure instead uses a sequence of thresholds: the smallest p-value is compared with the strictest threshold, and later p-values are compared with progressively less stringent thresholds if the earlier comparisons pass.

The procedure is useful because its guarantee does not require independence among tests. This matters in genomic settings, where tests can be dependent, for example because variants are correlated. The guarantee does, however, require valid p-values for the true null hypotheses and a clearly specified family of tests.

Definition: Suppose a family contains $m$ null hypotheses with p-values $p_1,\ldots,p_m$, and let $0<\alpha<1$ be the chosen significance level. Write the ordered p-values as $p_{(1)}\le p_{(2)}\le\cdots\le p_{(m)}$. Holm’s rank-$i$ threshold is $$ \frac{\alpha}{m-i+1},\qquad i=1,\ldots,m. $$ Starting at rank 1, reject the hypothesis corresponding to $p_{(i)}$ when $p_{(i)}\le \alpha/(m-i+1)$. Continue in order while each comparison passes. At the first rank $k$ that fails, do not reject that hypothesis or any hypothesis at a later rank.

The denominator counts how many hypotheses remain, including the one currently being tested. Thus, the first threshold is $\alpha/m$, the next is $\alpha/(m-1)$, and the last is $\alpha$. The threshold sequence is nondecreasing. A p-value is compared with the threshold for its ordered rank, not with a threshold assigned to a hypothesis before sorting.

Assumptions and Error Rate

For each true null hypothesis, its p-value must be valid: for every $t$ between 0 and 1, its probability of being at most $t$, when that null is true, is no greater than $t$. This is called super-uniformity. For a continuous test whose null p-value is exactly uniform, the probability is $t$; for a conservative or discrete test it can be smaller. No assumption of independence, positive dependence, or any other particular dependence structure among the p-values is needed.

Let $I_0$ be the set of indices of true null hypotheses in the family. The FWER is the probability of rejecting at least one of these true null hypotheses. Strong control at level $\alpha$ means that this probability is at most $\alpha$ for every configuration of true and false null hypotheses, not just when all null hypotheses are true. This is the error rate Holm’s procedure controls.

Worked Example: Applying the Ordered Thresholds

Consider five hypothetical variant-association tests at $\alpha=0.05$. Their ordered p-values are $0.003,\ 0.012,\ 0.018,\ 0.041,\ 0.40$. The tests are assumed to have valid p-values under their respective null hypotheses; no independence assumption is made.

There are $m=5$ tests. The ordered thresholds are $$ \frac{0.05}{5}=0.010,\quad \frac{0.05}{4}=0.0125,\quad \frac{0.05}{3}=0.0167,\quad \frac{0.05}{2}=0.025,\quad \frac{0.05}{1}=0.050. $$

At rank 1, $0.003\le0.010$, so reject the hypothesis with the smallest p-value. At rank 2, $0.012\le0.0125$, so reject that hypothesis as well. At rank 3, $0.018>0.0167$, so the procedure stops. It does not reject the rank-3 hypothesis or either of the remaining hypotheses, even though $0.041\le0.025$ is false and the final p-value is also above its threshold. The stopping rule means later comparisons cannot restart the procedure.

Why Strong FWER Control Holds

The key is to examine the smallest p-value among the true null hypotheses. This value is not necessarily the smallest p-value overall: false null hypotheses may have smaller p-values and appear earlier in the ordered list. But if Holm rejects any true null, it must have passed the rank of the smallest true-null p-value. That fact limits how small the smallest true-null p-value must be.

Theorem: If every true-null p-value is valid, Holm’s step-down procedure strongly controls the FWER at level $\alpha$, under arbitrary dependence among the p-values.

Locate the first true null in the ordering.
Let $m_0$ be the number of true null hypotheses, and let $k$ be the rank of the smallest p-value belonging to a true null. All $m_0$ true-null p-values occur at rank $k$ or later, so $m-k+1\ge m_0$.

Use the step-down rule.
If any true null is rejected, the procedure must have passed rank $k$. In particular, it must have rejected the true null at rank $k$, which requires $p_{(k)}\le\alpha/(m-k+1)\le\alpha/m_0$.

Bound the chance of that event.
The event of at least one false rejection is therefore contained in the event that at least one of the $m_0$ true-null p-values is at most $\alpha/m_0$. Apply validity and the union bound to obtain a probability no greater than $\alpha$.

In detail, label the true-null p-values $p_j$ for $j\in I_0$. The event inclusion from the argument above gives $$ \{\text{at least one true null is rejected}\} \subseteq \bigcup_{j\in I_0}\{p_j\le\alpha/m_0\}. $$ The union bound and validity of each true-null p-value then give $$ \operatorname{FWER} \le \sum_{j\in I_0}P(p_j\le\alpha/m_0) \le \sum_{j\in I_0}\frac{\alpha}{m_0} = \alpha. $$

This proof does not multiply probabilities or require joint independence. It only uses individual validity for the true-null p-values and the union bound, which applies regardless of their dependence. The argument also explains why the result is strong control: $m_0$ can be any number from 1 to $m$, corresponding to any configuration of true and false null hypotheses.

Worked Example: The Smallest True-Null Argument

Consider six hypothetical tests at $\alpha=0.05$. Suppose three null hypotheses are true, and the smallest p-value among those three is at rank $k=2$. For the procedure to reject any true null, it must pass rank 2. The threshold there is $$ \frac{0.05}{6-2+1}=\frac{0.05}{5}=0.010. $$

There are $m_0=3$ true nulls, so $m-k+1=5\ge3$, and the rank-2 threshold $0.010$ is no larger than $\alpha/m_0=0.05/3=0.0167$. Thus, a false rejection can occur only if at least one of the three true-null p-values is at most $0.0167$. By validity and the union bound, the probability of that event is at most $3(0.0167)$, which is approximately $0.050$. The p-values may be dependent; this bound remains valid.

Ties, Diagnostics, and Limitations

Tied p-values can be ordered in either order because their numerical values are equal. If a tied value passes at the earliest rank occupied by the tie, it also passes at the later ranks in that tied group, whose thresholds are at least as large. If it fails at the earliest such rank, the procedure stops before rejecting any of the tied hypotheses. Thus, the reject-or-not outcome for tied hypotheses does not depend on how their labels are ordered.

A practical diagnostic is to check that the complete testing family was fixed for the analysis and that the ordered p-values and their rank-specific thresholds are aligned correctly. Also check that tests have been conducted and calibrated as intended: Holm cannot repair invalid p-values caused by a misspecified model, an inappropriate test, or unaccounted-for selection. If a p-value is missing, silently dropping it can change $m$ and therefore every threshold; the handling of missing tests should be specified rather than improvised.

Holm protects against the chance of one or more false rejections, which can be a stringent objective when a genomic analysis contains many tests. It does not directly control the expected proportion of false discoveries among the rejected hypotheses. That is a different error-rate target. Nor does FWER control make an individual association estimate unbiased, establish biological importance, or show that an association is causal.

How Holm Relates to Bonferroni

Bonferroni compares every p-value with the same threshold, $\alpha/m$. Holm begins with that same threshold for the smallest p-value, then uses a new threshold at each successive rank. Both procedures control FWER without requiring independence when the p-values are valid. The operational distinction is that Bonferroni applies one cutoff to all tests, while Holm’s decision at later ranks depends on passing every earlier comparison. A failure in Holm’s sequence ends the rejection process; it is not appropriate to continue testing later p-values against their nominal rank thresholds.

Worked Example: A Tie at the Stopping Boundary

Suppose three hypothetical genomic tests have ordered p-values $0.010,\ 0.030,\ 0.030$, and $\alpha=0.05$. The thresholds are $0.05/3=0.0167$, $0.05/2=0.025$, and $0.05/1=0.050$. Assume all three p-values are valid under their true nulls.

The first comparison passes because $0.010\le0.0167$. The second fails because $0.030>0.025$, so the procedure stops and rejects neither hypothesis with p-value $0.030$. Although $0.030\le0.050$, the third comparison is not reached. Swapping the labels of the two tied tests cannot change the result: the first tied value encountered fails, and neither tied hypothesis is rejected.

Common Mistakes

Testing each p-value against its own rank threshold without stopping. Holm is a step-down rule. Once a comparison fails, no later hypothesis is rejected.
Sorting thresholds instead of p-values. Sort the p-values from smallest to largest, then pair ranks in order with $\alpha/m,\alpha/(m-1),\ldots,\alpha$.
Assuming independence is required. The strong-control proof uses valid true-null p-values and the union bound, not independence.
Calling a valid p-value proof of a valid analysis. The theorem assumes valid p-values under each true null; it does not establish that a chosen statistical model or testing family is appropriate.

Key takeaway: Holm’s procedure orders all $m$ p-values, tests sequentially against $\alpha/(m-i+1)$, and stops at the first failure. Its strong FWER guarantee follows because any false rejection requires the smallest true-null p-value to be at most $\alpha/m_0$, an event bounded by $\alpha$ using validity and the union bound.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For $m=4$ and $\alpha=0.05$, what are the four ordered Holm thresholds?
Why does rejecting a true null at rank $k$ require passing the comparison at the rank of the smallest true-null p-value?
Which assumption about dependence is needed for Holm’s strong FWER guarantee?
If the rank-2 comparison fails, may the rank-3 comparison still result in a rejection if its p-value is below the rank-3 threshold?
For $m_0=2$ true nulls, what is the union-bound upper bound for the probability that at least one true-null p-value is at most $\alpha/2$?

Solutions. (1) The thresholds are $0.05/4=0.0125,\ 0.05/3=0.0167,\ 0.05/2=0.025,$ and $0.05$. (2) Holm rejects sequentially and cannot pass a later rank without passing every earlier rank, including the rank of the smallest true-null p-value. (3) None is required; true-null p-values must be valid, but they may be arbitrarily dependent. (4) No. A failure ends the procedure, so rank 3 and all later ranks are not rejected. (5) The union bound is $2(\alpha/2)=\alpha$.

What You'll Learn