Holm Step-Down Family-Wise Error Control

Why Use a Step-Down Procedure?

A genome-wide analysis may test many variant–trait associations, while the scientific aim is to avoid making even one false discovery within a specified family of tests. The family-wise error rate (FWER) is the probability of rejecting at least one true null hypothesis in that family. A single common threshold can control this error, but it may be conservative when some hypotheses are false. Holm’s step-down procedure uses the ordering of the observed p-values to set a sequence of thresholds, while retaining FWER control.

The procedure begins with the smallest p-value and compares it with the strictest threshold. If it passes, the next p-value is compared with a less stringent threshold, and so on. It stops at the first failure: that hypothesis and every hypothesis after it are not rejected. This stopping rule is essential; the procedure is not a series of independent decisions in which a later p-value can be rejected after an earlier one fails.

Definition: Consider a family of \(m\) hypotheses with valid p-values \(p_1,\ldots,p_m\). Order them as \(p_{(1)}\le p_{(2)}\le\cdots\le p_{(m)}\), carrying each hypothesis label along with its p-value. At family-wise significance level \(\alpha\), Holm’s step-down procedure tests \(p_{(i)}\) against \(\alpha/(m-i+1)\), for \(i=1,\ldots,m\). Reject in sequence while the p-value is at most its threshold. At the first p-value greater than its threshold, stop and do not reject that hypothesis or any later one.

The threshold at rank \(i\) is the Bonferroni threshold \(\alpha/(m-i+1)\) for the hypotheses still under consideration. It grows as the procedure proceeds, so later p-values can pass a less stringent threshold. Holm’s procedure does not require independent tests. Its guarantee does require that, under each true null hypothesis, the corresponding p-value is valid: for every \(u\) between 0 and 1, \(P(p_j\le u\mid H_j\text{ is true})\le u\). This condition includes the usual continuous uniform-null case and allows conservative p-values.

Applying the Ordered Thresholds

The rank-specific thresholds are set by the total family size, not by how many hypotheses appear promising. If \(m=5\) and \(\alpha=0.05\), the ordered thresholds are \(0.05/5\), \(0.05/4\), \(0.05/3\), \(0.05/2\), and \(0.05\). A failure ends the sequence, even if a later p-value would be below its own larger threshold.

Worked Example: Finding the Rejection Set

In a hypothetical study, five association tests have ordered p-values \(0.001,\ 0.008,\ 0.014,\ 0.030,\ 0.200\). Apply Holm’s procedure at \(\alpha=0.05\).

Step 1. With \(m=5\), the thresholds are \(0.05/5=0.0100\), \(0.05/4=0.0125\), \(0.05/3=0.0167\), \(0.05/2=0.0250\), and \(0.05/1=0.0500\).

Step 2. Compare in order. The first p-value passes because \(0.001\le0.0100\). The second passes because \(0.008\le0.0125\). The third passes because \(0.014\le0.0167\). The fourth fails because \(0.030>0.0250\). The procedure stops at that failure; the fifth hypothesis is not rejected, even though \(0.200\) is greater than its threshold as well.

Conclusion. Holm rejects the hypotheses corresponding to the three smallest p-values. At the 0.05 family-wise level, these rejections are made with control of the probability of at least one false rejection across all five tests, provided the p-values are valid under their null hypotheses.

Holm-Adjusted P-Values

The ordered decisions can also be reported as adjusted p-values. These put each test on the same scale as the chosen family-wise significance level: a hypothesis is rejected at level \(\alpha\) if and only if its Holm-adjusted p-value is at most \(\alpha\). To calculate them, first multiply each ordered p-value by the number of hypotheses remaining at its rank. Then take the running maximum, so adjusted values cannot decrease as the ordered p-values increase. Finally, cap each value at 1.

p^{\mathrm{Holm}}_{(i)} = \min\left\{1,\ \max_{1\le j\le i}\left[(m-j+1)p_{(j)}\right]\right\}, \qquad i=1,\ldots,m.

The running maximum matters because the raw products \((m-i+1)p_{(i)}\) can decrease from one rank to the next. Without the maximum, these products would not necessarily give adjusted p-values consistent with the ordered step-down decisions. After calculating the adjusted values in rank order, attach each one to the hypothesis that had that p-value; do not reorder the labels independently.

Worked Example: Calculating Adjusted P-Values

Suppose five hypothetical association tests have ordered p-values \(0.004,\ 0.009,\ 0.011,\ 0.040,\ 0.700\). Calculate their Holm-adjusted p-values.

Step 1. Multiply each p-value by the number of hypotheses remaining. The raw products are \(5(0.004)=0.020\), \(4(0.009)=0.036\), \(3(0.011)=0.033\), \(2(0.040)=0.080\), and \(1(0.700)=0.700\).

Step 2. Take the running maximum and cap at 1. The first adjusted value is \(0.020\). The second is \(\max(0.020,0.036)=0.036\). The third is \(\max(0.020,0.036,0.033)=0.036\). The fourth is \(\max(0.020,0.036,0.033,0.080)=0.080\). The fifth is \(0.700\). None exceeds 1, so no further capping changes them.

Conclusion. In rank order, the adjusted p-values are \(0.020,\ 0.036,\ 0.036,\ 0.080,\ 0.700\). At \(\alpha=0.05\), the first two hypotheses are rejected. The third raw product, \(0.033\), is smaller than the second, but its adjusted value remains \(0.036\) because the running maximum preserves the required ordering.

Why the FWER Guarantee Holds Under Dependence

Holm’s guarantee is a strong FWER guarantee: it applies to any configuration of true and false null hypotheses, not just the case in which every null is true. The argument uses the validity of the individual p-values and the union bound. It does not assume that the p-values are independent, so it remains valid when association tests are dependent, as can occur for correlated variants.

Theorem: If every p-value is valid under its corresponding true null hypothesis, Holm’s step-down procedure controls the FWER at level \(\alpha\), for any dependence structure among the p-values.

To justify the result, let \(m_0\) be the number of true null hypotheses. If \(m_0=0\), there can be no false rejection, so the FWER is 0. Otherwise, suppose at least one true null is rejected, and let rank \(i\) be the first position in the ordered list occupied by a true null. Every hypothesis before rank \(i\) is then false. For a true null at rank \(i\) to be rejected, the procedure must have reached that rank and passed its comparison. Thus \(p_{(i)}\le\alpha/(m-i+1)\). Since there are \(m_0\) true nulls at or after rank \(i\), \(m_0\le m-i+1\), which gives \(\alpha/(m-i+1)\le\alpha/m_0\). The p-value \(p_{(i)}\) is the smallest p-value among the true nulls, so this event implies that at least one true-null p-value is at most \(\alpha/m_0\).

\begin{aligned} P(\text{at least one true null is rejected}) &\le P\left(\bigcup_{j\in\mathcal{T}}\{p_j\le\alpha/m_0\}\right)\\ &\le \sum_{j\in\mathcal{T}} P(p_j\le\alpha/m_0)\\ &\le m_0\left(\frac{\alpha}{m_0}\right)=\alpha, \end{aligned}

Here \(\mathcal{T}\) is the set of true null hypotheses. The second inequality is the union bound, which does not require independence. The final inequality follows from p-value validity for each true null. This proves control of the probability of one or more false rejections at no more than \(\alpha\).

Worked Example: Returning Adjusted Values to Variant Labels

In a hypothetical analysis, four variants have association p-values: variant A, \(0.018\); variant B, \(0.003\); variant C, \(0.041\); and variant D, \(0.009\). Calculate Holm-adjusted p-values and determine which variants are rejected at \(\alpha=0.05\).

Step 1. Sort the results while retaining their labels: B has \(p_{(1)}=0.003\), D has \(p_{(2)}=0.009\), A has \(p_{(3)}=0.018\), and C has \(p_{(4)}=0.041\). Here \(m=4\).

Step 2. The raw products are \(4(0.003)=0.012\), \(3(0.009)=0.027\), \(2(0.018)=0.036\), and \(1(0.041)=0.041\). They are already nondecreasing, so the running maxima give adjusted values \(0.012,\ 0.027,\ 0.036,\ 0.041\), respectively.

Step 3. Return the values to the original labels: B has adjusted p-value \(0.012\), D has \(0.027\), A has \(0.036\), and C has \(0.041\). Each is at most \(0.05\), so Holm rejects all four hypotheses at this level. The result is a family-wise decision, not a statement that each rejected association has a 95% probability of being true.

Defining the Family and Reading the Result

Holm’s calculation depends on \(m\), the number of hypotheses in the family. The family must therefore be specified for the scientific question before interpreting the adjusted results. In a genetic association analysis, the relevant family could be all variant–trait tests in a planned analysis or a narrower, prespecified set. Changing the family size changes the ranks’ thresholds and adjusted p-values; omitting tests after seeing their results does not preserve the intended error guarantee.

The method controls the chance of one or more false rejections, not the expected proportion of false findings among the rejections. That distinction is important when many associations are reported. Holm’s procedure can be less conservative than applying the single Bonferroni threshold \(\alpha/m\) to every test, because its later thresholds increase after earlier hypotheses pass. It still controls FWER under arbitrary dependence, whereas procedures that rely on independence require that additional assumption.

Common Mistakes

Continuing after a failure. A later p-value may be below its rank-specific threshold, but the step-down procedure has already stopped. No hypotheses at or after the first failure are rejected.
Using the wrong denominator. At rank \(i\), use \(m-i+1\), the number of hypotheses remaining in the ordered sequence, not \(i\) or a count chosen after inspecting the results.
Ignoring the running maximum. Raw adjusted-value products can decrease. Use the cumulative maximum, capped at 1, before interpreting them as Holm-adjusted p-values.
Assuming independence is required. Holm’s strong FWER guarantee follows from valid individual p-values and the union bound; it does not require independent association tests.
Changing the testing family after seeing the p-values. The family defines \(m\), so it must match the set of tests for which the error guarantee is claimed.

Key takeaway: Order the p-values, compare them sequentially with \(\alpha/(m-i+1)\), and stop at the first failure. Holm-adjusted p-values use a running maximum of the corresponding scaled p-values. With valid individual p-values, this procedure controls the family-wise error rate under arbitrary dependence.

Check Your Understanding

Use these questions to check the procedure and its interpretation.

For \(m=6\) tests at \(\alpha=0.05\), what threshold is used at rank 3?
Why must the Holm procedure stop after the first p-value that exceeds its threshold?
For ordered p-values \(0.002,\ 0.015,\ 0.030\) in a family of three tests, which hypotheses are rejected at \(\alpha=0.05\)? Show each comparison.
Why does the Holm FWER proof not require independent p-values?
What adjustment is needed if the raw scaled products decrease when calculating Holm-adjusted p-values?