Holm Adjustments for Multiple Tests

Why Adjust p-Values Across Multiple Tests?

When several hypotheses are tested in the same analysis, even tests that are all null can collectively create a substantial chance of at least one false positive. For example, after an ANOVA, a researcher might compare several pairs of group means. Each unadjusted p-value answers a question about one comparison, but interpreting many such tests together raises a family-level question: how likely is at least one incorrect rejection among them?

Holm’s procedure addresses that question by ordering the p-values and testing them in sequence. Its first threshold is as strict as the Bonferroni threshold for the full family; later thresholds become progressively less strict. The sequence stops at the first p-value that fails its threshold. This step-down rule controls the familywise error rate while often rejecting more hypotheses than applying the same Bonferroni threshold to every comparison.

Definition: A family is the set of hypotheses treated together for a multiple-testing adjustment. The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in that family. Holm’s procedure is a step-down method that compares the ordered p-values with thresholds determined by their ranks and the family size.

To use Holm’s method, first decide which hypotheses belong in the family and obtain one valid, unadjusted p-value for each. If there are \(m\) tests, write the ordered p-values as \(p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}\). The parentheses indicate rank in the ordered list, not the identity of a particular comparison. For a chosen familywise significance level \(\alpha\), compare each p-value with its Holm threshold.

\text{At rank } i,\quad p_{(i)} \leq \frac{\alpha}{m-i+1}

Begin at rank 1. If the p-value meets its threshold, reject the null hypothesis associated with it and continue to rank 2. Continue rejecting while each p-value meets its own threshold. At the first failure, stop: retain that null hypothesis and all hypotheses at later ranks. A later p-value is not tested for rejection after the sequence stops, even if it would meet its own threshold considered by itself.

Procedure: Order the \(m\) unadjusted p-values from smallest to largest. At rank \(i\), compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Reject in sequence for as long as each comparison passes. Stop at the first failure and do not reject any hypothesis at that rank or later.

Worked Examples

Worked Example: Applying the Stopping Rule

Suppose four hypothetical pairwise comparisons of group means produce unadjusted p-values \(0.004\), \(0.018\), \(0.020\), and \(0.030\). Apply Holm’s procedure at \(\alpha=0.05\). The values are already ordered from smallest to largest.

There are \(m=4\) tests. The thresholds, in order, are:

\frac{0.05}{4}=0.0125,\qquad \frac{0.05}{3}=0.01667,\qquad \frac{0.05}{2}=0.025,\qquad \frac{0.05}{1}=0.05.

At rank 1, \(0.004 \le 0.0125\), so reject the null hypothesis for the comparison with p-value \(0.004\). At rank 2, \(0.018 > 0.01667\), so the sequence stops. The Holm conclusion is to reject only the first null hypothesis. Do not reject the hypotheses with p-values \(0.018\), \(0.020\), or \(0.030\), even though the last two would be below their own rank-specific thresholds if those comparisons were reached.

The key feature is that the thresholds do not authorize separate, independent decisions: they are used in sequence, with stopping at the first failure.

Worked Example: Continuing Through the Whole Family

Five hypothetical tests have ordered p-values \(0.006\), \(0.011\), \(0.014\), \(0.018\), and \(0.030\). Apply Holm’s procedure at \(\alpha=0.05\).

Here \(m=5\). The ordered comparisons are:

\begin{aligned} 0.006 &\leq \frac{0.05}{5}=0.0100,\\ 0.011 &\leq \frac{0.05}{4}=0.0125,\\ 0.014 &\leq \frac{0.05}{3}=0.01667,\\ 0.018 &\leq \frac{0.05}{2}=0.0250,\\ 0.030 &\leq \frac{0.05}{1}=0.0500. \end{aligned}

Every comparison passes, so reject all five null hypotheses at the familywise level \(\alpha=0.05\). Each decision is made only after all earlier ranks have passed.

For comparison, applying the same Bonferroni cutoff to every test would require each p-value to be no greater than \(0.05/5=0.0100\), which only the first p-value meets. Holm’s method begins with that same strict cutoff, but relaxes the threshold as it proceeds. This example shows how Holm can reject more hypotheses than the single-cutoff Bonferroni method while still controlling the familywise error rate.

Worked Example: Mapping Decisions Back to Comparisons

A hypothetical study compares six pairs of treatment-group means. The unadjusted p-values are: A versus B, \(0.017\); A versus C, \(0.002\); A versus D, \(0.044\); B versus C, \(0.011\); B versus D, \(0.027\); and C versus D, \(0.008\). Apply Holm’s procedure at \(\alpha=0.05\), keeping track of which comparison belongs to each ordered p-value.

Sort the values and label their comparisons: rank 1 is A versus C, \(0.002\); rank 2 is C versus D, \(0.008\); rank 3 is B versus C, \(0.011\); rank 4 is A versus B, \(0.017\); rank 5 is B versus D, \(0.027\); and rank 6 is A versus D, \(0.044\). With \(m=6\), the first four thresholds are \(0.05/6=0.008333\), \(0.05/5=0.0100\), \(0.05/4=0.0125\), and \(0.05/3=0.01667\).

The first three tests pass: \(0.002 \le 0.008333\), \(0.008 \le 0.0100\), and \(0.011 \le 0.0125\). At rank 4, \(0.017 > 0.01667\), so stop. Reject the null hypotheses for A versus C, C versus D, and B versus C. Do not reject the null hypotheses for A versus B, B versus D, or A versus D. The decisions must be reported against the original comparisons, not just as ranks.

Holm-Adjusted p-Values

The same Holm decisions can be expressed using adjusted p-values. An adjusted p-value is the smallest familywise significance level at which that hypothesis would be rejected by the procedure. To calculate them, first multiply each ordered p-value by its rank-specific denominator, \(m-i+1\). Then take the running maximum of those products, and cap any result above 1 at 1. The running maximum ensures that adjusted p-values do not decrease as the ordered p-values increase.

p_{\mathrm{adj},(i)} = \min\left(1,\ \max_{1\leq j\leq i}\left[(m-j+1)p_{(j)}\right]\right)

A hypothesis is rejected at familywise level \(\alpha\) exactly when its Holm-adjusted p-value is no greater than \(\alpha\). The adjustments are calculated in sorted order, then attached back to the hypotheses that produced those p-values.

Worked Example: Calculating Adjusted p-Values

Use the five ordered p-values \(0.006\), \(0.011\), \(0.014\), \(0.018\), and \(0.030\) from the previous example. Since \(m=5\), the products of each p-value and its rank-specific denominator are:

5(0.006)=0.030,\quad 4(0.011)=0.044,\quad 3(0.014)=0.042,\quad 2(0.018)=0.036,\quad 1(0.030)=0.030.

Take the running maximum: \(0.030\), then \(\max(0.030,0.044)=0.044\), then \(\max(0.030,0.044,0.042)=0.044\), and the running maximum remains \(0.044\) for ranks 4 and 5. The adjusted p-values in sorted order are therefore \(0.030\), \(0.044\), \(0.044\), \(0.044\), and \(0.044\). All are below \(0.05\), matching the conclusion that all five hypotheses are rejected at that familywise level.

Why the Procedure Controls Familywise Error

Holm’s control of FWER follows from a short argument using the union bound. Suppose \(m_0\) of the \(m\) null hypotheses are true, and imagine that Holm’s procedure rejects at least one of those true null hypotheses. Consider the first true null hypothesis encountered in the ordered sequence, at rank \(k\). All \(m_0\) true null hypotheses must be at rank \(k\) or later. Thus \(m-k+1 \geq m_0\).

For Holm to reach and reject the true null at rank \(k\), its p-value must pass the threshold at that rank. Therefore:

p_{(k)} \leq \frac{\alpha}{m-k+1} \leq \frac{\alpha}{m_0}.

So, whenever at least one true null is rejected, at least one of the \(m_0\) true-null p-values is no greater than \(\alpha/m_0\). A valid p-value for a true null satisfies \(P(p_i \leq t) \leq t\) for every \(t\) between 0 and 1. By the union bound, the probability that any of the \(m_0\) true-null p-values is at most \(\alpha/m_0\) is no greater than:

\sum_{i=1}^{m_0} P\left(p_i \leq \frac{\alpha}{m_0}\right) \leq m_0\left(\frac{\alpha}{m_0}\right) =\alpha.

Consequently, the probability of rejecting at least one true null hypothesis is at most \(\alpha\). This argument does not require the tests to be independent. It does require valid p-values for true null hypotheses and a clearly defined family of tests.

Common Mistakes and Practical Interpretation

Using the same cutoff at every rank. Holm’s thresholds change with rank: the denominator decreases from \(m\) to 1. Using \(\alpha/m\) for all tests is the Bonferroni single-cutoff rule, not the Holm step-down procedure.
Continuing after a failure. Once an ordered p-value exceeds its threshold, stop. Later p-values are not rejected under Holm, even if an isolated comparison might appear small relative to a later threshold.
Sorting thresholds instead of p-values. Sort the p-values from smallest to largest, keeping each comparison attached to its p-value. Then compare in rank order.
Changing the family after seeing results. The family should reflect the set of tests being treated together, not a convenient subset chosen after inspecting the p-values. The family determines \(m\), and changing \(m\) can change the decisions.
Calling an adjusted p-value the probability a null hypothesis is true. It is not. It summarizes the evidence against a hypothesis under the multiple-testing procedure; it does not give the probability that the hypothesis is true.

In context, a Holm rejection supports evidence against the corresponding null hypothesis while controlling the chance of at least one false rejection across the specified family at level \(\alpha\). It does not establish that every rejected comparison is important in practice, nor does it turn an association into a causal effect. Report the comparisons, the family and its size, the chosen \(\alpha\), and the adjusted decisions clearly.

Key takeaway: Holm’s method orders the p-values, applies thresholds \(\alpha/m, \alpha/(m-1), \ldots, \alpha\) in sequence, and stops at the first failure. This step-down procedure controls familywise error and can be more powerful than applying one Bonferroni cutoff to every test.

Check Your Understanding

Try these questions without looking back.

Four ordered p-values are \(0.008\), \(0.014\), \(0.021\), and \(0.040\). At \(\alpha=0.05\), what is the first Holm threshold, and where does the procedure stop?
Why can a p-value at a later rank not be declared significant under Holm after an earlier rank fails?
For \(m=3\), write the three Holm thresholds in order at significance level \(\alpha\).
What does a Holm-adjusted p-value of \(0.032\) mean for decisions at familywise levels \(0.05\) and \(0.01\)?
Does Holm’s familywise error control require the tests to be independent? State the key condition on the p-values used in the control argument.